Oct 31 03:18:53.480501 ip-10-0-134-255 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 03:18:53.887851 ip-10-0-134-255 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:18:53.887851 ip-10-0-134-255 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 03:18:53.887851 ip-10-0-134-255 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:18:53.887851 ip-10-0-134-255 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:18:53.891245 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891164    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:18:53.891245 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891243    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:18:53.891245 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891247    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:18:53.891335 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891250    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:18:53.891335 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891254    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:18:53.891335 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891258    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:18:53.891335 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891262    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:18:53.891335 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891264    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:18:53.891335 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891268    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:18:53.891335 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891270    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:18:53.891335 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891279    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:18:53.891335 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891282    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:18:53.891335 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891284    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:18:53.891335 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891287    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:18:53.891335 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891290    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:18:53.891335 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891294    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:18:53.891335 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891296    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:18:53.891335 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891299    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:18:53.891335 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891301    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:18:53.891335 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891304    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:18:53.891335 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891306    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:18:53.891335 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891309    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:18:53.891769 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891311    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:18:53.891769 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891314    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:18:53.891769 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891317    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:18:53.891769 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891319    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:18:53.891769 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891322    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:18:53.891769 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891325    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:18:53.891769 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891327    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:18:53.891769 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891330    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:18:53.891769 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891332    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:18:53.891769 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891335    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:18:53.891769 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891338    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:18:53.891769 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891340    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:18:53.891769 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891343    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:18:53.891769 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891346    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:18:53.891769 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891349    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:18:53.891769 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891351    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:18:53.891769 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891354    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:18:53.891769 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891357    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:18:53.891769 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891360    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:18:53.892234 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891362    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:18:53.892234 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891365    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:18:53.892234 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891367    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:18:53.892234 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891370    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:18:53.892234 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891372    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:18:53.892234 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891374    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:18:53.892234 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891377    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:18:53.892234 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891380    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:18:53.892234 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891382    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:18:53.892234 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891385    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:18:53.892234 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891388    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:18:53.892234 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891391    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:18:53.892234 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891393    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:18:53.892234 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891396    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:18:53.892234 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891400    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:18:53.892234 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891402    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:18:53.892234 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891405    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:18:53.892234 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891408    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:18:53.892234 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891410    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:18:53.892676 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891413    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:18:53.892676 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891419    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:18:53.892676 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891424    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:18:53.892676 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891427    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:18:53.892676 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891429    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:18:53.892676 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891433    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:18:53.892676 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891436    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:18:53.892676 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891439    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:18:53.892676 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891442    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:18:53.892676 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891445    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:18:53.892676 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891447    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:18:53.892676 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891450    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:18:53.892676 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891454    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:18:53.892676 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891457    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:18:53.892676 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891459    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:18:53.892676 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891462    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:18:53.892676 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891464    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:18:53.892676 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891467    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:18:53.892676 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891469    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:18:53.893162 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891472    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:18:53.893162 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891474    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:18:53.893162 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891477    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:18:53.893162 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891479    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:18:53.893162 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891482    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:18:53.893162 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891485    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:18:53.893162 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891487    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:18:53.893162 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891490    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:18:53.893162 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891493    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:18:53.893162 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891498    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:18:53.893162 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891502    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:18:53.893162 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891506    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:18:53.893162 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891509    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:18:53.893162 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891511    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:18:53.893162 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891514    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:18:53.893162 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891516    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:18:53.893162 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891519    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:18:53.893162 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891522    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:18:53.893162 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891524    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:18:53.893162 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891528    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:18:53.893650 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891531    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:18:53.893650 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891533    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:18:53.893650 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891536    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:18:53.893650 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891538    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:18:53.893650 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891542    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:18:53.893650 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891544    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:18:53.893650 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.891547    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:18:53.893650 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892358    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:18:53.893650 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892366    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:18:53.893650 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892369    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:18:53.893650 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892372    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:18:53.893650 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892375    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:18:53.893650 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892378    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:18:53.893650 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892380    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:18:53.893650 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892383    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:18:53.893650 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892386    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:18:53.893650 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892388    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:18:53.893650 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892391    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:18:53.893650 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892393    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:18:53.893650 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892396    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:18:53.894156 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892399    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:18:53.894156 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892402    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:18:53.894156 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892405    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:18:53.894156 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892407    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:18:53.894156 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892410    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:18:53.894156 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892413    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:18:53.894156 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892415    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:18:53.894156 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892418    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:18:53.894156 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892421    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:18:53.894156 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892424    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:18:53.894156 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892426    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:18:53.894156 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892429    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:18:53.894156 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892431    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:18:53.894156 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892434    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:18:53.894156 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892437    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:18:53.894156 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892439    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:18:53.894156 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892442    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:18:53.894156 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892444    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:18:53.894156 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892447    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:18:53.894607 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892450    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:18:53.894607 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892453    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:18:53.894607 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892455    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:18:53.894607 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892457    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:18:53.894607 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892460    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:18:53.894607 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892462    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:18:53.894607 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892465    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:18:53.894607 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892469    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:18:53.894607 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892473    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:18:53.894607 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892476    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:18:53.894607 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892479    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:18:53.894607 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892482    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:18:53.894607 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892485    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:18:53.894607 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892488    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:18:53.894607 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892491    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:18:53.894607 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892494    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:18:53.894607 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892496    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:18:53.894607 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892500    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:18:53.894607 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892502    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:18:53.895063 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892505    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:18:53.895063 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892507    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:18:53.895063 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892510    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:18:53.895063 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892512    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:18:53.895063 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892515    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:18:53.895063 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892518    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:18:53.895063 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892520    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:18:53.895063 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892522    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:18:53.895063 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892525    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:18:53.895063 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892528    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:18:53.895063 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892530    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:18:53.895063 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892532    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:18:53.895063 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892535    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:18:53.895063 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892538    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:18:53.895063 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892542    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:18:53.895063 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892544    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:18:53.895063 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892547    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:18:53.895063 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892549    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:18:53.895063 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892552    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:18:53.895063 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892554    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:18:53.895546 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892557    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:18:53.895546 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892559    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:18:53.895546 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892562    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:18:53.895546 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892564    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:18:53.895546 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892567    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:18:53.895546 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892571    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:18:53.895546 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892575    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:18:53.895546 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892578    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:18:53.895546 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892581    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:18:53.895546 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892584    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:18:53.895546 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892586    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:18:53.895546 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892589    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:18:53.895546 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892592    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:18:53.895546 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892594    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:18:53.895546 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892596    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:18:53.895546 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892599    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:18:53.895546 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892601    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:18:53.895546 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892604    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:18:53.895546 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892607    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:18:53.895989 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892609    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:18:53.895989 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892612    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:18:53.895989 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892615    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:18:53.895989 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892618    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:18:53.895989 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892620    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:18:53.895989 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892623    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:18:53.895989 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892625    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:18:53.895989 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892628    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:18:53.895989 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892637    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:18:53.895989 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892640    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:18:53.895989 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892643    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:18:53.895989 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892645    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:18:53.895989 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892648    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:18:53.895989 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892651    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:18:53.895989 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892653    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:18:53.895989 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.892656    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:18:53.895989 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.893978    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:18:53.895989 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.893984    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:18:53.895989 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.893988    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:18:53.895989 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.893992    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:18:53.896478 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.893995    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:18:53.896478 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.893997    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:18:53.896478 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894000    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:18:53.896478 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894002    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:18:53.896478 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894005    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:18:53.896478 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894008    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:18:53.896478 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894010    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:18:53.896478 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894013    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:18:53.896478 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894015    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:18:53.896478 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894018    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:18:53.896478 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894020    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:18:53.896478 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894023    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:18:53.896478 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894026    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:18:53.896478 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894031    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:18:53.896478 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894035    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:18:53.896478 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894038    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:18:53.896478 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894041    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:18:53.896478 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894043    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:18:53.896478 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894046    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:18:53.896971 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894048    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:18:53.896971 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894051    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:18:53.896971 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894054    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:18:53.896971 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894056    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:18:53.896971 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894059    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:18:53.896971 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894061    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:18:53.896971 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894064    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:18:53.896971 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894066    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:18:53.896971 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894069    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:18:53.896971 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894071    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:18:53.896971 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894073    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:18:53.896971 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894076    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:18:53.896971 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894079    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:18:53.896971 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894082    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:18:53.896971 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894084    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:18:53.896971 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894087    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:18:53.896971 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894107    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:18:53.896971 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894111    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:18:53.896971 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894114    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:18:53.897456 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894117    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:18:53.897456 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894119    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:18:53.897456 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894122    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:18:53.897456 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894124    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:18:53.897456 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894127    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:18:53.897456 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894130    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:18:53.897456 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894132    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:18:53.897456 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894135    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:18:53.897456 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894138    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:18:53.897456 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894141    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:18:53.897456 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894144    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:18:53.897456 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894147    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:18:53.897456 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894149    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:18:53.897456 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894152    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:18:53.897456 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894154    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:18:53.897456 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894157    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:18:53.897456 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894160    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:18:53.897456 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894163    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:18:53.897456 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894166    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:18:53.897456 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894169    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:18:53.897945 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894171    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:18:53.897945 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894174    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:18:53.897945 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894176    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:18:53.897945 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894179    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:18:53.897945 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894182    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:18:53.897945 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894184    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:18:53.897945 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894187    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:18:53.897945 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894190    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:18:53.897945 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894192    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:18:53.897945 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894195    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:18:53.897945 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894198    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:18:53.897945 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894200    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:18:53.897945 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894203    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:18:53.897945 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894205    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:18:53.897945 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894207    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:18:53.897945 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894210    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:18:53.897945 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894213    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:18:53.897945 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894215    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:18:53.897945 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894218    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:18:53.897945 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894220    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:18:53.898467 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894223    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:18:53.898467 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894225    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:18:53.898467 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894228    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:18:53.898467 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894230    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:18:53.898467 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894233    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:18:53.898467 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894235    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:18:53.898467 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894238    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:18:53.898467 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894240    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:18:53.898467 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894243    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:18:53.898467 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894246    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:18:53.898467 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894248    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:18:53.898467 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894250    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:18:53.898467 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894254    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:18:53.898467 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894256    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:18:53.898467 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894258    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:18:53.898467 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894261    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:18:53.898467 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894264    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:18:53.898467 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894267    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:18:53.898467 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894269    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:18:53.898913 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894272    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:18:53.898913 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894275    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:18:53.898913 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894277    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:18:53.898913 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894279    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:18:53.898913 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.894282    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:18:53.898913 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894379    2577 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 03:18:53.898913 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894387    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 03:18:53.898913 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894393    2577 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 03:18:53.898913 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894398    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 03:18:53.898913 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894402    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 03:18:53.898913 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894406    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 03:18:53.898913 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894410    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 03:18:53.898913 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894414    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 03:18:53.898913 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894418    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 03:18:53.898913 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894421    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 03:18:53.898913 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894424    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 03:18:53.898913 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894427    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 03:18:53.898913 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894430    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 03:18:53.898913 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894433    2577 flags.go:64] FLAG: --cgroup-root=""
Oct 31 03:18:53.898913 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894436    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 03:18:53.898913 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894439    2577 flags.go:64] FLAG: --client-ca-file=""
Oct 31 03:18:53.898913 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894442    2577 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 03:18:53.898913 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894446    2577 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 03:18:53.898913 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894450    2577 flags.go:64] FLAG: --cluster-domain=""
Oct 31 03:18:53.899500 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894453    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 03:18:53.899500 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894456    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 03:18:53.899500 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894459    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 03:18:53.899500 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894463    2577 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 03:18:53.899500 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894470    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 03:18:53.899500 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894474    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 03:18:53.899500 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894477    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 03:18:53.899500 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894480    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 03:18:53.899500 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894483    2577 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 03:18:53.899500 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894486    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 03:18:53.899500 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894488    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 03:18:53.899500 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894492    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 03:18:53.899500 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894495    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 03:18:53.899500 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894499    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 03:18:53.899500 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894501    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 03:18:53.899500 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894504    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 03:18:53.899500 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894507    2577 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 03:18:53.899500 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894510    2577 flags.go:64] FLAG: --enable-server="true"
Oct 31 03:18:53.899500 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894513    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 03:18:53.899500 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894517    2577 flags.go:64] FLAG: --event-burst="100"
Oct 31 03:18:53.899500 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894520    2577 flags.go:64] FLAG: --event-qps="50"
Oct 31 03:18:53.899500 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894525    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 03:18:53.899500 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894528    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 03:18:53.899500 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894531    2577 flags.go:64] FLAG: --eviction-hard=""
Oct 31 03:18:53.900131 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894536    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 03:18:53.900131 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894539    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 03:18:53.900131 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894542    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 03:18:53.900131 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894545    2577 flags.go:64] FLAG: --eviction-soft=""
Oct 31 03:18:53.900131 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894548    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 03:18:53.900131 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894551    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 03:18:53.900131 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894554    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 03:18:53.900131 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894557    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 03:18:53.900131 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894560    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 03:18:53.900131 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894563    2577 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 03:18:53.900131 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894566    2577 flags.go:64] FLAG: --feature-gates=""
Oct 31 03:18:53.900131 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894570    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 03:18:53.900131 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894574    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 03:18:53.900131 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894578    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 03:18:53.900131 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894581    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 03:18:53.900131 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894584    2577 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 03:18:53.900131 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894587    2577 flags.go:64] FLAG: --help="false"
Oct 31 03:18:53.900131 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894590    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:18:53.900131 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894593    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 03:18:53.900131 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894596    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 03:18:53.900131 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894599    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 03:18:53.900131 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894603    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 03:18:53.900131 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894606    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894609    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894612    2577 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894615    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894618    2577 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894621    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894624    2577 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894628    2577 flags.go:64] FLAG: --kube-reserved=""
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894631    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894634    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894637    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894640    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894643    2577 flags.go:64] FLAG: --lock-file=""
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894646    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894649    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894652    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894658    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894661    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894664    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894667    2577 flags.go:64] FLAG: --logging-format="text"
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894670    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894674    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894678    2577 flags.go:64] FLAG: --manifest-url=""
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894681    2577 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894686    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 03:18:53.900788 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894689    2577 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894693    2577 flags.go:64] FLAG: --max-pods="110"
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894696    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894699    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894702    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894705    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894708    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894711    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894714    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894722    2577 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894725    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894728    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894731    2577 flags.go:64] FLAG: --pod-cidr=""
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894734    2577 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894738    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894741    2577 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894744    2577 flags.go:64] FLAG: --port="10250"
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894747    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894750    2577 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0864006630e1e7726"
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894753    2577 flags.go:64] FLAG: --qos-reserved=""
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894755    2577 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894758    2577 flags.go:64] FLAG: --register-node="true"
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894761    2577 flags.go:64] FLAG: --register-with-taints=""
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894765    2577 flags.go:64] FLAG: --registry-burst="10"
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894768    2577 flags.go:64] FLAG: --registry-qps="5"
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894771    2577 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 03:18:53.901439 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894775    2577 flags.go:64] FLAG: --reserved-memory=""
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894779    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894782    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894786    2577 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894789    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894791    2577 flags.go:64] FLAG: --runonce="false"
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894795    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894798    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894801    2577 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894803    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894806    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894809    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894812    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894815    2577 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894818    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894821    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894824    2577 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894827    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894830    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894833    2577 flags.go:64] FLAG: --system-cgroups=""
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894837    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894842    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894845    2577 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894848    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894857    2577 flags.go:64] FLAG: --tls-min-version=""
Oct 31 03:18:53.902064 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894860    2577 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 03:18:53.902687 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894862    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 03:18:53.902687 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894865    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 03:18:53.902687 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894868    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 03:18:53.902687 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894871    2577 flags.go:64] FLAG: --v="2"
Oct 31 03:18:53.902687 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894876    2577 flags.go:64] FLAG: --version="false"
Oct 31 03:18:53.902687 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894880    2577 flags.go:64] FLAG: --vmodule=""
Oct 31 03:18:53.902687 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894884    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 03:18:53.902687 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.894888    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 03:18:53.902687 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895022    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:18:53.902687 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895026    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:18:53.902687 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895029    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:18:53.902687 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895032    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:18:53.902687 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895035    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:18:53.902687 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895038    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:18:53.902687 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895040    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:18:53.902687 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895043    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:18:53.902687 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895046    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:18:53.902687 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895048    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:18:53.902687 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895051    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:18:53.902687 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895053    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:18:53.902687 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895056    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:18:53.903209 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895058    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:18:53.903209 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895061    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:18:53.903209 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895064    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:18:53.903209 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895066    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:18:53.903209 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895069    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:18:53.903209 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895073    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:18:53.903209 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895077    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:18:53.903209 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895081    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:18:53.903209 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895085    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:18:53.903209 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895104    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:18:53.903209 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895108    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:18:53.903209 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895111    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:18:53.903209 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895113    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:18:53.903209 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895116    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:18:53.903209 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895118    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:18:53.903209 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895121    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:18:53.903209 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895124    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:18:53.903209 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895126    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:18:53.903209 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895129    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:18:53.903690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895132    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:18:53.903690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895136    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:18:53.903690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895139    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:18:53.903690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895142    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:18:53.903690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895145    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:18:53.903690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895147    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:18:53.903690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895150    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:18:53.903690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895152    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:18:53.903690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895155    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:18:53.903690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895157    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:18:53.903690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895160    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:18:53.903690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895163    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:18:53.903690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895165    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:18:53.903690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895168    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:18:53.903690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895170    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:18:53.903690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895172    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:18:53.903690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895175    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:18:53.903690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895178    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:18:53.903690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895181    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:18:53.903690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895184    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:18:53.904192 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895187    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:18:53.904192 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895190    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:18:53.904192 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895192    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:18:53.904192 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895195    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:18:53.904192 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895197    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:18:53.904192 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895199    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:18:53.904192 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895202    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:18:53.904192 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895204    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:18:53.904192 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895207    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:18:53.904192 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895209    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:18:53.904192 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895212    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:18:53.904192 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895214    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:18:53.904192 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895217    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:18:53.904192 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895221    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:18:53.904192 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895224    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:18:53.904192 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895226    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:18:53.904192 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895229    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:18:53.904192 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895232    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:18:53.904192 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895234    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:18:53.904637 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895237    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:18:53.904637 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895239    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:18:53.904637 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895242    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:18:53.904637 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895244    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:18:53.904637 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895248    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:18:53.904637 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895252    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:18:53.904637 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895255    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:18:53.904637 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895258    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:18:53.904637 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895261    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:18:53.904637 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895264    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:18:53.904637 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895266    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:18:53.904637 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895272    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:18:53.904637 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895275    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:18:53.904637 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895277    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:18:53.904637 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895280    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:18:53.904637 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895282    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:18:53.904637 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895285    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:18:53.904637 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895287    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:18:53.904637 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895290    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:18:53.905116 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895292    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:18:53.905116 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895295    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:18:53.905116 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895297    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:18:53.905116 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895300    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:18:53.905116 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895302    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:18:53.905116 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895305    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:18:53.905116 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895307    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:18:53.905116 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895311    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:18:53.905116 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895314    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:18:53.905116 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895317    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:18:53.905116 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895319    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:18:53.905116 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895322    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:18:53.905116 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895324    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:18:53.905116 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895327    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:18:53.905116 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895329    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:18:53.905116 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895332    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:18:53.905506 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.895358    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:18:53.905506 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895395    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:18:53.905506 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895399    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:18:53.905506 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895402    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:18:53.905506 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895404    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:18:53.905506 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895408    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:18:53.905506 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895412    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:18:53.905506 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895415    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:18:53.905506 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895419    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:18:53.905506 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895422    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:18:53.905506 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895425    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:18:53.905506 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895427    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:18:53.905506 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895430    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:18:53.905506 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895432    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:18:53.905506 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895435    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:18:53.905506 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895437    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:18:53.905894 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895440    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:18:53.905894 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895443    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:18:53.905894 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895445    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:18:53.905894 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895448    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:18:53.905894 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895450    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:18:53.905894 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895453    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:18:53.905894 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895455    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:18:53.905894 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895457    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:18:53.905894 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895461    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:18:53.905894 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895464    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:18:53.905894 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895467    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:18:53.905894 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895470    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:18:53.905894 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895472    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:18:53.905894 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895475    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:18:53.905894 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895478    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:18:53.905894 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895481    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:18:53.905894 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895484    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:18:53.905894 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895486    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:18:53.905894 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895488    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:18:53.905894 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895491    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:18:53.906391 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895494    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:18:53.906391 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895496    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:18:53.906391 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895499    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:18:53.906391 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895501    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:18:53.906391 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895505    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:18:53.906391 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895508    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:18:53.906391 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895510    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:18:53.906391 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895513    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:18:53.906391 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895515    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:18:53.906391 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895518    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:18:53.906391 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895520    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:18:53.906391 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895523    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:18:53.906391 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895525    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:18:53.906391 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895528    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:18:53.906391 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895530    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:18:53.906391 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895533    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:18:53.906391 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895535    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:18:53.906391 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895538    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:18:53.906391 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895541    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:18:53.906845 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895543    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:18:53.906845 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895547    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:18:53.906845 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895550    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:18:53.906845 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895553    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:18:53.906845 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895556    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:18:53.906845 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895559    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:18:53.906845 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895562    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:18:53.906845 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895566    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:18:53.906845 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895568    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:18:53.906845 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895571    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:18:53.906845 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895574    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:18:53.906845 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895576    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:18:53.906845 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895579    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:18:53.906845 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895581    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:18:53.906845 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895584    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:18:53.906845 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895586    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:18:53.906845 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895589    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:18:53.906845 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895592    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:18:53.906845 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895595    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:18:53.907323 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895598    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:18:53.907323 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895600    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:18:53.907323 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895603    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:18:53.907323 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895605    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:18:53.907323 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895608    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:18:53.907323 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895610    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:18:53.907323 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895613    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:18:53.907323 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895615    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:18:53.907323 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895618    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:18:53.907323 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895620    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:18:53.907323 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895623    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:18:53.907323 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895625    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:18:53.907323 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895628    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:18:53.907323 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895630    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:18:53.907323 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895634    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:18:53.907323 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895637    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:18:53.907323 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895639    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:18:53.907323 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895642    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:18:53.907323 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895645    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:18:53.907323 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895647    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:18:53.907917 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895650    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:18:53.907917 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895652    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:18:53.907917 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895655    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:18:53.907917 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895658    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:18:53.907917 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895661    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:18:53.907917 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895664    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:18:53.907917 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895666    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:18:53.907917 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895669    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:18:53.907917 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895671    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:18:53.907917 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895674    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:18:53.907917 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895677    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:18:53.907917 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895680    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:18:53.907917 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.895682    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:18:53.907917 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.903082    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 03:18:53.907917 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.903117    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 03:18:53.907917 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.903870    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 03:18:53.907917 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.903877    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 03:18:53.907917 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.903938    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:18:53.907917 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.903945    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:18:53.908453 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.903950    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:18:53.908453 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.903955    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:18:53.908453 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.903958    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:18:53.908453 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.903961    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:18:53.908453 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.903964    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:18:53.908453 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.903968    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:18:53.908453 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.903970    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:18:53.908453 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.903973    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:18:53.908453 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.903976    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:18:53.908453 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.903979    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:18:53.908453 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.903982    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:18:53.908453 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.903985    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:18:53.908453 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.903988    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:18:53.908453 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.903991    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:18:53.908453 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.903993    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:18:53.908453 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.903996    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:18:53.908453 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.903999    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:18:53.908453 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904001    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:18:53.908453 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904004    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:18:53.909177 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904007    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:18:53.909177 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904010    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:18:53.909177 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904012    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:18:53.909177 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904015    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:18:53.909177 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904018    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:18:53.909177 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904020    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:18:53.909177 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904024    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:18:53.909177 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904026    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:18:53.909177 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904031    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:18:53.909177 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904033    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:18:53.909177 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904036    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:18:53.909177 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904039    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:18:53.909177 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904041    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:18:53.909177 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904044    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:18:53.909177 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904047    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:18:53.909177 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904050    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:18:53.909177 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904052    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:18:53.909177 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904055    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:18:53.909177 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904058    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:18:53.909690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904060    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:18:53.909690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904063    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:18:53.909690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904065    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:18:53.909690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904068    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:18:53.909690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904070    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:18:53.909690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904073    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:18:53.909690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904075    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:18:53.909690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904078    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:18:53.909690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904081    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:18:53.909690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904084    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:18:53.909690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904087    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:18:53.909690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904105    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:18:53.909690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904109    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:18:53.909690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904112    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:18:53.909690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904115    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:18:53.909690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904118    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:18:53.909690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904120    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:18:53.909690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904124    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:18:53.909690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904127    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:18:53.909690 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904129    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:18:53.910360 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904132    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:18:53.910360 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904136    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:18:53.910360 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904139    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:18:53.910360 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904142    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:18:53.910360 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904145    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:18:53.910360 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904148    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:18:53.910360 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904150    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:18:53.910360 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904153    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:18:53.910360 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904156    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:18:53.910360 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904159    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:18:53.910360 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904161    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:18:53.910360 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904164    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:18:53.910360 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904167    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:18:53.910360 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904169    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:18:53.910360 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904172    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:18:53.910360 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904174    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:18:53.910360 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904177    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:18:53.910360 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904179    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:18:53.910360 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904182    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:18:53.910860 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904184    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:18:53.910860 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904187    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:18:53.910860 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904190    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:18:53.910860 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904193    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:18:53.910860 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904197    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:18:53.910860 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904200    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:18:53.910860 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904203    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:18:53.910860 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904206    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:18:53.910860 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904209    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:18:53.910860 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904211    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:18:53.910860 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904214    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:18:53.910860 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904217    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:18:53.910860 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904219    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:18:53.910860 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904222    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:18:53.910860 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904226    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:18:53.910860 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904228    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:18:53.910860 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904232    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:18:53.910860 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904235    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:18:53.910860 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904237    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:18:53.910860 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904240    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:18:53.911343 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904243    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:18:53.911343 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904245    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:18:53.911343 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904248    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:18:53.911343 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904250    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:18:53.911343 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904252    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:18:53.911343 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904255    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:18:53.911343 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904257    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:18:53.911343 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.904287    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:18:53.911343 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904428    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:18:53.911343 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904433    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:18:53.911343 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904437    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:18:53.911343 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904440    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:18:53.911343 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904442    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:18:53.911343 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904445    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:18:53.911343 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904448    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:18:53.911343 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904450    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:18:53.911774 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904453    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:18:53.911774 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904456    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:18:53.911774 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904458    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:18:53.911774 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904461    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:18:53.911774 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904463    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:18:53.911774 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904466    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:18:53.911774 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904468    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:18:53.911774 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904471    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:18:53.911774 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904473    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:18:53.911774 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904476    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:18:53.911774 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904479    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:18:53.911774 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904482    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:18:53.911774 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904485    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:18:53.911774 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904488    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:18:53.911774 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904491    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:18:53.911774 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904493    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:18:53.911774 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904496    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:18:53.911774 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904498    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:18:53.911774 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904501    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:18:53.911774 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904503    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:18:53.912296 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904506    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:18:53.912296 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904508    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:18:53.912296 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904511    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:18:53.912296 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904513    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:18:53.912296 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904517    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:18:53.912296 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904523    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:18:53.912296 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904526    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:18:53.912296 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904529    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:18:53.912296 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904531    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:18:53.912296 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904534    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:18:53.912296 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904536    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:18:53.912296 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904539    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:18:53.912296 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904542    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:18:53.912296 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904544    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:18:53.912296 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904547    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:18:53.912296 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904550    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:18:53.912296 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904553    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:18:53.912296 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904555    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:18:53.912296 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904558    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:18:53.912787 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904560    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:18:53.912787 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904563    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:18:53.912787 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904566    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:18:53.912787 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904569    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:18:53.912787 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904572    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:18:53.912787 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904574    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:18:53.912787 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904577    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:18:53.912787 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904580    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:18:53.912787 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904583    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:18:53.912787 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904585    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:18:53.912787 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904588    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:18:53.912787 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904591    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:18:53.912787 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904593    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:18:53.912787 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904596    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:18:53.912787 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904598    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:18:53.912787 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904601    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:18:53.912787 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904604    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:18:53.912787 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904606    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:18:53.912787 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904608    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:18:53.912787 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904611    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:18:53.913300 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904614    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:18:53.913300 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904616    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:18:53.913300 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904619    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:18:53.913300 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904621    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:18:53.913300 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904624    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:18:53.913300 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904626    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:18:53.913300 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904628    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:18:53.913300 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904631    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:18:53.913300 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904634    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:18:53.913300 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904636    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:18:53.913300 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904639    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:18:53.913300 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904642    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:18:53.913300 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904644    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:18:53.913300 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904651    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:18:53.913300 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904653    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:18:53.913300 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904656    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:18:53.913300 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904659    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:18:53.913300 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904661    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:18:53.913300 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904664    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:18:53.913752 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904667    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:18:53.913752 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904670    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:18:53.913752 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904673    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:18:53.913752 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904676    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:18:53.913752 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904679    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:18:53.913752 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904681    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:18:53.913752 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904683    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:18:53.913752 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904686    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:18:53.913752 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904689    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:18:53.913752 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904691    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:18:53.913752 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904694    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:18:53.913752 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904696    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:18:53.913752 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904699    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:18:53.913752 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904701    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:18:53.913752 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904704    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:18:53.913752 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904706    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:18:53.913752 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904709    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:18:53.913752 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904712    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:18:53.913752 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904714    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:18:53.913752 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904717    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:18:53.914351 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.904740    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:18:53.914351 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904773    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:18:53.914351 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904776    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:18:53.914351 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904779    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:18:53.914351 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904782    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:18:53.914351 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904785    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:18:53.914351 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904787    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:18:53.914351 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904791    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:18:53.914351 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904794    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:18:53.914351 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904797    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:18:53.914351 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904799    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:18:53.914351 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904802    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:18:53.914351 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904805    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:18:53.914351 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904807    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:18:53.914351 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904811    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:18:53.914707 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904813    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:18:53.914707 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904816    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:18:53.914707 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904818    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:18:53.914707 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904821    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:18:53.914707 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904823    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:18:53.914707 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904826    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:18:53.914707 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904829    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:18:53.914707 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904831    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:18:53.914707 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904834    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:18:53.914707 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904837    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:18:53.914707 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904839    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:18:53.914707 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904842    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:18:53.914707 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904844    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:18:53.914707 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904847    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:18:53.914707 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904849    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:18:53.914707 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904852    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:18:53.914707 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904855    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:18:53.914707 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904857    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:18:53.914707 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904860    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:18:53.914707 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904862    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:18:53.915212 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904864    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:18:53.915212 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904867    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:18:53.915212 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904869    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:18:53.915212 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904872    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:18:53.915212 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904876    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:18:53.915212 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904879    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:18:53.915212 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904881    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:18:53.915212 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904884    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:18:53.915212 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904887    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:18:53.915212 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904891    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:18:53.915212 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904894    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:18:53.915212 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904897    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:18:53.915212 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904900    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:18:53.915212 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904903    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:18:53.915212 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904906    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:18:53.915212 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904909    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:18:53.915212 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904911    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:18:53.915212 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904914    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:18:53.915212 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904916    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:18:53.915212 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904919    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:18:53.915728 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904923    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:18:53.915728 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904926    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:18:53.915728 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904928    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:18:53.915728 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904931    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:18:53.915728 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904933    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:18:53.915728 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904937    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:18:53.915728 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904939    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:18:53.915728 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904942    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:18:53.915728 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904945    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:18:53.915728 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904947    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:18:53.915728 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904950    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:18:53.915728 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904952    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:18:53.915728 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904955    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:18:53.915728 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904957    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:18:53.915728 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904960    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:18:53.915728 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904962    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:18:53.915728 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904966    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:18:53.915728 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904969    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:18:53.915728 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904971    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:18:53.915728 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904974    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:18:53.916315 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904978    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:18:53.916315 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904981    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:18:53.916315 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904983    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:18:53.916315 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904986    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:18:53.916315 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904988    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:18:53.916315 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904991    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:18:53.916315 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904994    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:18:53.916315 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904996    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:18:53.916315 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.904999    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:18:53.916315 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.905001    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:18:53.916315 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.905004    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:18:53.916315 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.905007    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:18:53.916315 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.905011    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:18:53.916315 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.905014    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:18:53.916315 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.905016    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:18:53.916315 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.905019    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:18:53.916315 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.905022    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:18:53.916315 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.905024    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:18:53.916315 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.905027    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:18:53.916791 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.905030    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:18:53.916791 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.905032    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:18:53.916791 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.905035    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:18:53.916791 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.905037    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:18:53.916791 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.905040    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:18:53.916791 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.905043    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:18:53.916791 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.905045    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:18:53.916791 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.905048    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:18:53.916791 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.905050    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:18:53.916791 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.905054    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:18:53.916791 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.905056    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:18:53.916791 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.905060    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:18:53.916791 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:53.905063    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:18:53.916791 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.905156    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 03:18:53.916791 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.907488    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 03:18:53.916791 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.908763    2577 server.go:1014] "Starting client certificate rotation"
Oct 31 03:18:53.916791 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.908873    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 03:18:53.916791 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.909688    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 03:18:53.917328 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.913853    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 03:18:53.917328 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.915222    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 03:18:53.924985 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.924967    2577 log.go:25] "Validated CRI v1 runtime API"
Oct 31 03:18:53.929031 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.929013    2577 log.go:25] "Validated CRI v1 image API"
Oct 31 03:18:53.929918 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.929892    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 03:18:53.934698 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.934667    2577 fs.go:135] Filesystem UUIDs: map[5da6bae3-0e3d-45a2-b121-c50b9b3927ca:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 93f112c0-26c5-439b-8707-25e3a4b2478c:/dev/nvme0n1p4]
Oct 31 03:18:53.934808 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.934695    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 03:18:53.940083 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.939952    2577 manager.go:217] Machine: {Timestamp:2026-10-31 03:18:53.938533771 +0000 UTC m=+0.349812447 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106627 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec214fd7e717df96091d645188425bcf SystemUUID:ec214fd7-e717-df96-091d-645188425bcf BootID:9f8e5106-2c1b-490b-bc47-9d7a2a1370b8 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:69:28:50:19:8f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:69:28:50:19:8f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:03:47:d5:ea:bf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 03:18:53.940083 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.940075    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 03:18:53.940214 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.940189    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 03:18:53.943458 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.943431    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 03:18:53.943598 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.943459    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-255.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 03:18:53.943649 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.943601    2577 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 03:18:53.943649 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.943609    2577 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 03:18:53.943649 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.943636    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 03:18:53.943725 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.943651    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 03:18:53.944892 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.944880    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 03:18:53.949186 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.949170    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 03:18:53.949320 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.949310    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 03:18:53.950119 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.950109    2577 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 03:18:53.950164 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.950123    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 03:18:53.950164 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.950133    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 03:18:53.950164 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.950142    2577 kubelet.go:405] "Adding apiserver pod source"
Oct 31 03:18:53.950164 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.950151    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 03:18:53.950511 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.950499    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 03:18:53.953196 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.953180    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 03:18:53.962433 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.962412    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 03:18:53.963226 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.963214    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 03:18:53.963266 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.963245    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 03:18:53.963433 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.963423    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 03:18:53.963489 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.963440    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 03:18:53.963489 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.963446    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 03:18:53.963489 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.963451    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 03:18:53.963489 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.963457    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 03:18:53.963489 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.963463    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 03:18:53.964151 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.964142    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 03:18:53.964151 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.964151    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 03:18:53.964213 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.964159    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 03:18:53.964213 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.964165    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 03:18:53.964213 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.964173    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 03:18:53.964213 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.964182    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 03:18:53.964916 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.964906    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 03:18:53.964916 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.964916    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 03:18:53.968787 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.968772    2577 server.go:1263] "Started kubelet"
Oct 31 03:18:53.969067 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.968995    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 03:18:53.969161 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.969075    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 03:18:53.969161 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.968850    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 03:18:53.969614 ip-10-0-134-255 systemd[1]: Started Kubernetes Kubelet.
Oct 31 03:18:53.971520 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.971505    2577 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 03:18:53.973058 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.972604    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 03:18:53.975396 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:53.975379    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 03:18:53.978146 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.978128    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 03:18:53.978687 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.978671    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 03:18:53.978820 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:53.978804    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-255.us-west-1.compute.internal\" not found"
Oct 31 03:18:53.979498 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.979483    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 03:18:53.979568 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.979484    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 03:18:53.979568 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.979510    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 03:18:53.979732 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.979714    2577 factory.go:153] Registering CRI-O factory
Oct 31 03:18:53.979822 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.979736    2577 factory.go:223] Registration of the crio container factory successfully
Oct 31 03:18:53.979822 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.979731    2577 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 03:18:53.979822 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.979788    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 03:18:53.979822 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.979797    2577 factory.go:55] Registering systemd factory
Oct 31 03:18:53.979822 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.979803    2577 factory.go:223] Registration of the systemd container factory successfully
Oct 31 03:18:53.979822 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.979815    2577 factory.go:103] Registering Raw factory
Oct 31 03:18:53.979822 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.979824    2577 manager.go:1199] Started watching for new ooms in manager
Oct 31 03:18:53.980032 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.979790    2577 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 03:18:53.980781 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.980767    2577 manager.go:319] Starting recovery of all containers
Oct 31 03:18:53.990521 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.990479    2577 manager.go:324] Recovery completed
Oct 31 03:18:53.995737 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.995721    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:18:53.997331 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.997313    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:18:53.997401 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.997344    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:18:53.997401 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.997355    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:18:53.997897 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.997882    2577 cpu_manager.go:226] "Starting" policy="none"
Oct 31 03:18:53.997897 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.997894    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 03:18:53.997990 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.997910    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 03:18:53.997990 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.997935    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:18:53.998631 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.998615    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:18:53.998688 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.998645    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:18:53.998688 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.998657    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:18:53.999999 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:53.999986    2577 policy_none.go:50] "Start"
Oct 31 03:18:54.000047 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.000004    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 03:18:54.000047 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.000014    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 03:18:54.001831 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.001820    2577 policy_none.go:44] "Start"
Oct 31 03:18:54.043748 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.043714    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 03:18:54.048387 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.048373    2577 manager.go:341] "Starting Device Plugin manager"
Oct 31 03:18:54.073590 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:54.049035    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 03:18:54.073590 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.049051    2577 server.go:87] "Starting device plugin registration server"
Oct 31 03:18:54.073590 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.049341    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 03:18:54.073590 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.049496    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 03:18:54.073590 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.049656    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 03:18:54.073590 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.049784    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 03:18:54.073590 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.049794    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 03:18:54.073590 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:54.050357    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 03:18:54.073590 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:54.050589    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-255.us-west-1.compute.internal\" not found"
Oct 31 03:18:54.073590 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.068775    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 03:18:54.073590 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.070124    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 03:18:54.073590 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.070143    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 03:18:54.073590 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.070163    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 03:18:54.073590 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:54.070195    2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 03:18:54.111556 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:54.111527    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-255.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 03:18:54.111556 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.111547    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-255.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 03:18:54.112672 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:54.111526    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-255.us-west-1.compute.internal.18e37f048272bf3c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-255.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-255.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 03:18:53.968744252 +0000 UTC m=+0.380022777,LastTimestamp:2026-10-31 03:18:53.968744252 +0000 UTC m=+0.380022777,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-255.us-west-1.compute.internal,}"
Oct 31 03:18:54.149843 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.149781    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:18:54.150697 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.150679    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:18:54.150794 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.150711    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:18:54.150794 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.150728    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:18:54.150794 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.150752    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:18:54.171514 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.171495    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal"]
Oct 31 03:18:54.171602 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.171548    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:18:54.172401 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.172387    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:18:54.172478 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.172413    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:18:54.172478 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.172423    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:18:54.174492 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.174480    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:18:54.174651 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.174635    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:18:54.174741 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.174670    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:18:54.175215 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.175198    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:18:54.175306 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.175230    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:18:54.175306 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.175241    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:18:54.175306 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.175197    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:18:54.175444 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.175313    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:18:54.175444 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.175331    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:18:54.177437 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.177422    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:18:54.177485 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.177463    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:18:54.178477 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.178455    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:18:54.178569 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.178484    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:18:54.178569 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.178500    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:18:54.180813 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.180780    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d21c869370d277ce1b45bad9863021dc-config\") pod \"kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal\" (UID: \"d21c869370d277ce1b45bad9863021dc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:18:54.180890 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.180825    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2075c7dc615e009d2cc914067d053547-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal\" (UID: \"2075c7dc615e009d2cc914067d053547\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:18:54.180890 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.180847    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2075c7dc615e009d2cc914067d053547-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal\" (UID: \"2075c7dc615e009d2cc914067d053547\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:18:54.195268 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:54.195250    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-255.us-west-1.compute.internal\" not found" node="ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:18:54.197719 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:54.197704    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-255.us-west-1.compute.internal\" not found" node="ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:18:54.202103 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.202079    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 03:18:54.202163 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.202153    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sxddq"
Oct 31 03:18:54.271388 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.271346    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:18:54.273148 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.273131    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:18:54.281836 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.281814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d21c869370d277ce1b45bad9863021dc-config\") pod \"kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal\" (UID: \"d21c869370d277ce1b45bad9863021dc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:18:54.281836 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.281828    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d21c869370d277ce1b45bad9863021dc-config\") pod \"kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal\" (UID: \"d21c869370d277ce1b45bad9863021dc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:18:54.281944 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.281853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2075c7dc615e009d2cc914067d053547-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal\" (UID: \"2075c7dc615e009d2cc914067d053547\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:18:54.281944 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.281871    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2075c7dc615e009d2cc914067d053547-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal\" (UID: \"2075c7dc615e009d2cc914067d053547\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:18:54.281944 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.281906    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2075c7dc615e009d2cc914067d053547-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal\" (UID: \"2075c7dc615e009d2cc914067d053547\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:18:54.281944 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.281939    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2075c7dc615e009d2cc914067d053547-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal\" (UID: \"2075c7dc615e009d2cc914067d053547\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:18:54.371018 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.370993    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:18:54.372065 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.372045    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:18:54.372223 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.372206    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:18:54.379509 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.379492    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:18:54.440902 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.440820    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:18:54.441714 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.441700    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:18:54.499649 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.499625    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:18:54.499921 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.499905    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:18:54.504716 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.504701    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:18:54.908542 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.908459    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 03:18:54.909047 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.908629    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 03:18:54.909047 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.908642    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 03:18:54.909047 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:54.908598    2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a3bc042c2ecd546b6b1e906ab067c5a8-f96d67daba3dd493.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-134-255.us-west-1.compute.internal.18e37f0484276074\": read tcp 10.0.134.255:35994->3.128.47.91:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-134-255.us-west-1.compute.internal.18e37f0484276074  default   5034 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-255.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-255.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-255.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 03:18:53 +0000 UTC,LastTimestamp:2026-10-31 03:18:54.150732491 +0000 UTC m=+0.562011016,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-255.us-west-1.compute.internal,}"
Oct 31 03:18:54.909047 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.908677    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 03:18:54.950504 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.950476    2577 apiserver.go:52] "Watching apiserver"
Oct 31 03:18:54.978874 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:54.978841    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 03:18:55.209353 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.209244    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:13:55 +0000 UTC" deadline="2028-06-06 18:54:24.670580171 +0000 UTC"
Oct 31 03:18:55.209353 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.209325    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14031h35m29.461288671s"
Oct 31 03:18:55.217875 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.217847    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 03:18:55.218272 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.218254    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-rtp6w","openshift-ovn-kubernetes/ovnkube-node-rkkr4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-bx98q","kube-system/kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp","openshift-image-registry/node-ca-2kzdk","kube-system/konnectivity-agent-kqgxt","openshift-dns/node-resolver-j86qh","openshift-multus/multus-additional-cni-plugins-lkj57","openshift-network-operator/iptables-alerter-9x2kw","kube-system/global-pull-secret-syncer-7nb9d","openshift-cluster-node-tuning-operator/tuned-l494z","openshift-multus/multus-vxkwg"]
Oct 31 03:18:55.220663 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.220644    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:18:55.220746 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:55.220724    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtp6w" podUID="bf1633f9-bdb5-4a26-bd80-a83a4b5587e8"
Oct 31 03:18:55.222713 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.222696    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.225359 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.225327    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:18:55.225471 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:55.225399    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx98q" podUID="45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9"
Oct 31 03:18:55.227417 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.227400    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:18:55.229505 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.229487    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2kzdk"
Oct 31 03:18:55.231605 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.231592    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kqgxt"
Oct 31 03:18:55.233677 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.233661    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j86qh"
Oct 31 03:18:55.235778 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.235765    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:18:55.237932 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.237914    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9x2kw"
Oct 31 03:18:55.240050 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.240033    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:18:55.240147 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:55.240117    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nb9d" podUID="8d251d9b-c5e7-46e7-b5ab-524b8171d83b"
Oct 31 03:18:55.242299 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.242281    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.244473 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.244455    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.276128 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.275460    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 03:18:55.276128 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.275567    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 03:18:55.276128 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.275639    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 03:18:55.276128 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.275774    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sv7k2\""
Oct 31 03:18:55.276128 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.275784    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 03:18:55.276128 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.275787    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 03:18:55.276128 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.275968    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 03:18:55.279855 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.279821    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 03:18:55.279855 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.279836    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 03:18:55.280062 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.280049    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 03:18:55.280148 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.280137    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4kwqf\""
Oct 31 03:18:55.280538 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.280523    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 03:18:55.281687 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.281652    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ltpjd\""
Oct 31 03:18:55.281904 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.281802    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 03:18:55.281904 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.281837    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 03:18:55.281904 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.281841    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 03:18:55.283772 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.283757    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 03:18:55.283856 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.283832    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 03:18:55.283908 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.283891    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xmnbh\""
Oct 31 03:18:55.285724 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.285705    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 03:18:55.289426 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.289403    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-var-lib-kubelet\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.289519 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.289437    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-registration-dir\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:18:55.289587 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.289524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-log-socket\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.289587 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.289545    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs\") pod \"network-metrics-daemon-bx98q\" (UID: \"45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9\") " pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:18:55.289587 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.289560    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 03:18:55.289587 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.289563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8da6593d-d284-438d-96ee-ec0d9ddf1124-serviceca\") pod \"node-ca-2kzdk\" (UID: \"8da6593d-d284-438d-96ee-ec0d9ddf1124\") " pod="openshift-image-registry/node-ca-2kzdk"
Oct 31 03:18:55.289757 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.289602    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-systemd\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.289757 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.289627    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-host\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.289757 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.289650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-cnibin\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.289757 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.289674    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-run-k8s-cni-cncf-io\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.289757 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.289712    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5jxx\" (UniqueName: \"kubernetes.io/projected/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8-kube-api-access-p5jxx\") pod \"network-check-target-rtp6w\" (UID: \"bf1633f9-bdb5-4a26-bd80-a83a4b5587e8\") " pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:18:55.289757 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.289744    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:18:55.289969 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.289768    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8da6593d-d284-438d-96ee-ec0d9ddf1124-host\") pod \"node-ca-2kzdk\" (UID: \"8da6593d-d284-438d-96ee-ec0d9ddf1124\") " pod="openshift-image-registry/node-ca-2kzdk"
Oct 31 03:18:55.289969 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.289791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8ppp\" (UniqueName: \"kubernetes.io/projected/8da6593d-d284-438d-96ee-ec0d9ddf1124-kube-api-access-s8ppp\") pod \"node-ca-2kzdk\" (UID: \"8da6593d-d284-438d-96ee-ec0d9ddf1124\") " pod="openshift-image-registry/node-ca-2kzdk"
Oct 31 03:18:55.289969 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.289807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-original-pull-secret\") pod \"global-pull-secret-syncer-7nb9d\" (UID: \"8d251d9b-c5e7-46e7-b5ab-524b8171d83b\") " pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:18:55.289969 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.289821    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-kubernetes\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.289969 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.289836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-tuned\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.289969 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.289875    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-multus-cni-dir\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.289969 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.289900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74face88-7a89-4b8f-9e8b-adea29761ed8-cni-binary-copy\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.289969 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.289922    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/74face88-7a89-4b8f-9e8b-adea29761ed8-multus-daemon-config\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.289969 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.289943    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-device-dir\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:18:55.289969 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.289966    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:18:55.290333 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.289994    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-multus-conf-dir\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.290333 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-run-multus-certs\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.290333 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290040    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-etc-selinux\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:18:55.290333 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290056    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-kubelet\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.290333 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290076    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-run-ovn-kubernetes\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.290333 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290121    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ff0da67d-c75e-4487-ba6a-b85e04a496d9-konnectivity-ca\") pod \"konnectivity-agent-kqgxt\" (UID: \"ff0da67d-c75e-4487-ba6a-b85e04a496d9\") " pod="kube-system/konnectivity-agent-kqgxt"
Oct 31 03:18:55.290333 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290143    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-dbus\") pod \"global-pull-secret-syncer-7nb9d\" (UID: \"8d251d9b-c5e7-46e7-b5ab-524b8171d83b\") " pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:18:55.290333 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290160    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-run-netns\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.290333 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290181    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-slash\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.290333 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-cni-bin\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.290333 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290224    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ff0da67d-c75e-4487-ba6a-b85e04a496d9-agent-certs\") pod \"konnectivity-agent-kqgxt\" (UID: \"ff0da67d-c75e-4487-ba6a-b85e04a496d9\") " pod="kube-system/konnectivity-agent-kqgxt"
Oct 31 03:18:55.290333 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290273    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce-hosts-file\") pod \"node-resolver-j86qh\" (UID: \"65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce\") " pod="openshift-dns/node-resolver-j86qh"
Oct 31 03:18:55.290333 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290293    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:18:55.290333 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290310    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.290333 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-run\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.290834 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290351    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4wv2\" (UniqueName: \"kubernetes.io/projected/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-kube-api-access-n4wv2\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.290834 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290380    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-multus-socket-dir-parent\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.290834 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290403    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce-tmp-dir\") pod \"node-resolver-j86qh\" (UID: \"65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce\") " pod="openshift-dns/node-resolver-j86qh"
Oct 31 03:18:55.290834 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-lib-modules\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.290834 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-var-lib-cni-multus\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.290834 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290477    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2jf8\" (UniqueName: \"kubernetes.io/projected/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-kube-api-access-b2jf8\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:18:55.290834 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2aa25e13-aaa4-4795-a221-2056b76e8c8f-host-slash\") pod \"iptables-alerter-9x2kw\" (UID: \"2aa25e13-aaa4-4795-a221-2056b76e8c8f\") " pod="openshift-network-operator/iptables-alerter-9x2kw"
Oct 31 03:18:55.290834 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290530    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-run-systemd\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.290834 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290561    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-etc-openvswitch\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.290834 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-run-openvswitch\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.290834 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290614    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-socket-dir\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:18:55.290834 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-run-ovn\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.290834 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-cni-netd\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.290834 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290674    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j92zc\" (UniqueName: \"kubernetes.io/projected/65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce-kube-api-access-j92zc\") pod \"node-resolver-j86qh\" (UID: \"65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce\") " pod="openshift-dns/node-resolver-j86qh"
Oct 31 03:18:55.290834 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290704    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-kubelet-config\") pod \"global-pull-secret-syncer-7nb9d\" (UID: \"8d251d9b-c5e7-46e7-b5ab-524b8171d83b\") " pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:18:55.290834 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290745    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-sys\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.291286 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290772    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-tmp\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.291286 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290792    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-var-lib-cni-bin\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.291286 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290806    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-var-lib-kubelet\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.291286 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290829    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-run-netns\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.291286 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290843    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-node-log\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.291286 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290860    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-sys-fs\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:18:55.291286 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290888    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-cnibin\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:18:55.291286 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290914    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-modprobe-d\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.291286 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290931    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-system-cni-dir\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.291286 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.290976    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-hostroot\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.291286 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.291006    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d72ff8d-4480-4266-ab8b-a225f4245836-ovnkube-script-lib\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.291286 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.291028    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-system-cni-dir\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:18:55.291286 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.291044    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-cni-binary-copy\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:18:55.291286 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.291072    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-sysconfig\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.291286 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.291106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-sysctl-conf\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.291286 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.291130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2aa25e13-aaa4-4795-a221-2056b76e8c8f-iptables-alerter-script\") pod \"iptables-alerter-9x2kw\" (UID: \"2aa25e13-aaa4-4795-a221-2056b76e8c8f\") " pod="openshift-network-operator/iptables-alerter-9x2kw"
Oct 31 03:18:55.291712 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.291150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d72ff8d-4480-4266-ab8b-a225f4245836-ovn-node-metrics-cert\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.291712 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.291173    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjkmq\" (UniqueName: \"kubernetes.io/projected/7d72ff8d-4480-4266-ab8b-a225f4245836-kube-api-access-jjkmq\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.291712 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.291195    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:18:55.291712 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.291222    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-sysctl-d\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.291712 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.291242    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d72ff8d-4480-4266-ab8b-a225f4245836-ovnkube-config\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.291712 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.291287    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zzzr\" (UniqueName: \"kubernetes.io/projected/74face88-7a89-4b8f-9e8b-adea29761ed8-kube-api-access-5zzzr\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.291712 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.291313    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp4mc\" (UniqueName: \"kubernetes.io/projected/ebb8e08d-8243-492f-aef5-d3b6590d9f47-kube-api-access-cp4mc\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:18:55.291712 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.291331    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d72ff8d-4480-4266-ab8b-a225f4245836-env-overrides\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.291712 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.291353    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-etc-kubernetes\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.291712 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.291389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-systemd-units\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.291712 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.291406    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tpl7\" (UniqueName: \"kubernetes.io/projected/2aa25e13-aaa4-4795-a221-2056b76e8c8f-kube-api-access-6tpl7\") pod \"iptables-alerter-9x2kw\" (UID: \"2aa25e13-aaa4-4795-a221-2056b76e8c8f\") " pod="openshift-network-operator/iptables-alerter-9x2kw"
Oct 31 03:18:55.291712 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.291420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-var-lib-openvswitch\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.291712 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.291441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hjs5\" (UniqueName: \"kubernetes.io/projected/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-kube-api-access-2hjs5\") pod \"network-metrics-daemon-bx98q\" (UID: \"45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9\") " pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:18:55.371456 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.371422    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 03:18:55.371581 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.371498    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zgpg4"
Oct 31 03:18:55.375924 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.375904    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 03:18:55.376035 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.375949    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 03:18:55.376110 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.376030    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 03:18:55.376110 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.376038    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xz7gs\""
Oct 31 03:18:55.376110 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.376078    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 03:18:55.376110 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.376086    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 03:18:55.376267 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.376139    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:18:55.376267 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.376148    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:18:55.376267 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.376164    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 03:18:55.376267 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.376190    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 03:18:55.376267 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.376243    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 03:18:55.376440 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.376325    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4wd7m\""
Oct 31 03:18:55.376440 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.376373    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-b4ln2\""
Oct 31 03:18:55.376440 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.376376    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 03:18:55.376440 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.376394    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-67ksm\""
Oct 31 03:18:55.376440 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.376400    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-9x6rv\""
Oct 31 03:18:55.391856 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.391829    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d72ff8d-4480-4266-ab8b-a225f4245836-env-overrides\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.391985 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.391864    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-etc-kubernetes\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.391985 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.391884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-systemd-units\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.391985 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.391934    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tpl7\" (UniqueName: \"kubernetes.io/projected/2aa25e13-aaa4-4795-a221-2056b76e8c8f-kube-api-access-6tpl7\") pod \"iptables-alerter-9x2kw\" (UID: \"2aa25e13-aaa4-4795-a221-2056b76e8c8f\") " pod="openshift-network-operator/iptables-alerter-9x2kw"
Oct 31 03:18:55.391985 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.391938    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-etc-kubernetes\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.391985 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.391959    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-var-lib-openvswitch\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.391985 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.391980    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-systemd-units\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.391985 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.391983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hjs5\" (UniqueName: \"kubernetes.io/projected/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-kube-api-access-2hjs5\") pod \"network-metrics-daemon-bx98q\" (UID: \"45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9\") " pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:18:55.392373 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-var-lib-kubelet\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.392373 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392022    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-var-lib-openvswitch\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.392373 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392030    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-registration-dir\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:18:55.392373 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-log-socket\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.392373 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392083    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-var-lib-kubelet\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.392373 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392134    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-registration-dir\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:18:55.392373 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392142    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-log-socket\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.392373 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs\") pod \"network-metrics-daemon-bx98q\" (UID: \"45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9\") " pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:18:55.392373 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392188    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8da6593d-d284-438d-96ee-ec0d9ddf1124-serviceca\") pod \"node-ca-2kzdk\" (UID: \"8da6593d-d284-438d-96ee-ec0d9ddf1124\") " pod="openshift-image-registry/node-ca-2kzdk"
Oct 31 03:18:55.392373 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:55.392232    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:18:55.392373 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-systemd\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.392373 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392301    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-host\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.392373 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392310    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-systemd\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.392373 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:55.392319    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs podName:45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9 nodeName:}" failed. No retries permitted until 2026-10-31 03:18:55.892277422 +0000 UTC m=+2.303555949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs") pod "network-metrics-daemon-bx98q" (UID: "45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:18:55.392373 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392335    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-host\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.392373 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392340    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d72ff8d-4480-4266-ab8b-a225f4245836-env-overrides\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.392373 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392346    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-cnibin\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.393187 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392403    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-cnibin\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.393187 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-run-k8s-cni-cncf-io\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.393187 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392427    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-run-k8s-cni-cncf-io\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.393187 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392444    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5jxx\" (UniqueName: \"kubernetes.io/projected/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8-kube-api-access-p5jxx\") pod \"network-check-target-rtp6w\" (UID: \"bf1633f9-bdb5-4a26-bd80-a83a4b5587e8\") " pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:18:55.393187 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:18:55.393187 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392508    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8da6593d-d284-438d-96ee-ec0d9ddf1124-host\") pod \"node-ca-2kzdk\" (UID: \"8da6593d-d284-438d-96ee-ec0d9ddf1124\") " pod="openshift-image-registry/node-ca-2kzdk"
Oct 31 03:18:55.393187 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8ppp\" (UniqueName: \"kubernetes.io/projected/8da6593d-d284-438d-96ee-ec0d9ddf1124-kube-api-access-s8ppp\") pod \"node-ca-2kzdk\" (UID: \"8da6593d-d284-438d-96ee-ec0d9ddf1124\") " pod="openshift-image-registry/node-ca-2kzdk"
Oct 31 03:18:55.393187 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392564    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-original-pull-secret\") pod \"global-pull-secret-syncer-7nb9d\" (UID: \"8d251d9b-c5e7-46e7-b5ab-524b8171d83b\") " pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:18:55.393187 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-kubernetes\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.393187 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392582    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:18:55.393187 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392603    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8da6593d-d284-438d-96ee-ec0d9ddf1124-serviceca\") pod \"node-ca-2kzdk\" (UID: \"8da6593d-d284-438d-96ee-ec0d9ddf1124\") " pod="openshift-image-registry/node-ca-2kzdk"
Oct 31 03:18:55.393187 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392613    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-tuned\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.393187 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392635    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-multus-cni-dir\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.393187 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392658    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74face88-7a89-4b8f-9e8b-adea29761ed8-cni-binary-copy\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.393187 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392682    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-kubernetes\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.393187 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392683    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/74face88-7a89-4b8f-9e8b-adea29761ed8-multus-daemon-config\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.393187 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-device-dir\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:18:55.393951 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:18:55.393951 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392779    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-multus-conf-dir\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.393951 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392806    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-run-multus-certs\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.393951 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392829    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-etc-selinux\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:18:55.393951 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-kubelet\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.393951 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392874    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 03:18:55.393951 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392917    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-run-ovn-kubernetes\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.393951 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392951    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-device-dir\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:18:55.393951 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392959    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-multus-cni-dir\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.393951 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:55.393205    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:18:55.393951 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393247    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/74face88-7a89-4b8f-9e8b-adea29761ed8-multus-daemon-config\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.393951 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:55.393284    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-original-pull-secret podName:8d251d9b-c5e7-46e7-b5ab-524b8171d83b nodeName:}" failed. No retries permitted until 2026-10-31 03:18:55.893265483 +0000 UTC m=+2.304544028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-original-pull-secret") pod "global-pull-secret-syncer-7nb9d" (UID: "8d251d9b-c5e7-46e7-b5ab-524b8171d83b") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:18:55.393951 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393347    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-multus-conf-dir\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.393951 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393350    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-run-multus-certs\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.393951 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393392    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-kubelet\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.393951 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393401    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8da6593d-d284-438d-96ee-ec0d9ddf1124-host\") pod \"node-ca-2kzdk\" (UID: \"8da6593d-d284-438d-96ee-ec0d9ddf1124\") " pod="openshift-image-registry/node-ca-2kzdk"
Oct 31 03:18:55.393951 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393430    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-etc-selinux\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:18:55.394810 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.392880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-run-ovn-kubernetes\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.394810 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393470    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:18:55.394810 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ff0da67d-c75e-4487-ba6a-b85e04a496d9-konnectivity-ca\") pod \"konnectivity-agent-kqgxt\" (UID: \"ff0da67d-c75e-4487-ba6a-b85e04a496d9\") " pod="kube-system/konnectivity-agent-kqgxt"
Oct 31 03:18:55.394810 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-dbus\") pod \"global-pull-secret-syncer-7nb9d\" (UID: \"8d251d9b-c5e7-46e7-b5ab-524b8171d83b\") " pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:18:55.394810 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393549    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-run-netns\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.394810 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393572    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-slash\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.394810 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393611    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-run-netns\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.394810 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393619    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-slash\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.394810 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-cni-bin\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.394810 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393667    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ff0da67d-c75e-4487-ba6a-b85e04a496d9-agent-certs\") pod \"konnectivity-agent-kqgxt\" (UID: \"ff0da67d-c75e-4487-ba6a-b85e04a496d9\") " pod="kube-system/konnectivity-agent-kqgxt"
Oct 31 03:18:55.394810 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce-hosts-file\") pod \"node-resolver-j86qh\" (UID: \"65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce\") " pod="openshift-dns/node-resolver-j86qh"
Oct 31 03:18:55.394810 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393710    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:18:55.394810 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393689    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-cni-bin\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.394810 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.394810 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-run\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.394810 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4wv2\" (UniqueName: \"kubernetes.io/projected/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-kube-api-access-n4wv2\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.394810 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393835    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:18:55.395642 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393849    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-multus-socket-dir-parent\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.395642 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393866    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-run\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.395642 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393876    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce-tmp-dir\") pod \"node-resolver-j86qh\" (UID: \"65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce\") " pod="openshift-dns/node-resolver-j86qh"
Oct 31 03:18:55.395642 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393885    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74face88-7a89-4b8f-9e8b-adea29761ed8-cni-binary-copy\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.395642 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-lib-modules\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.395642 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393889    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.395642 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-var-lib-cni-multus\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.395642 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2jf8\" (UniqueName: \"kubernetes.io/projected/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-kube-api-access-b2jf8\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:18:55.395642 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2aa25e13-aaa4-4795-a221-2056b76e8c8f-host-slash\") pod \"iptables-alerter-9x2kw\" (UID: \"2aa25e13-aaa4-4795-a221-2056b76e8c8f\") " pod="openshift-network-operator/iptables-alerter-9x2kw"
Oct 31 03:18:55.395642 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.393998    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-run-systemd\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.395642 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-etc-openvswitch\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.395642 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394042    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ff0da67d-c75e-4487-ba6a-b85e04a496d9-konnectivity-ca\") pod \"konnectivity-agent-kqgxt\" (UID: \"ff0da67d-c75e-4487-ba6a-b85e04a496d9\") " pod="kube-system/konnectivity-agent-kqgxt"
Oct 31 03:18:55.395642 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394113    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-var-lib-cni-multus\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.395642 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394120    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-run-openvswitch\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.395642 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394166    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce-tmp-dir\") pod \"node-resolver-j86qh\" (UID: \"65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce\") " pod="openshift-dns/node-resolver-j86qh"
Oct 31 03:18:55.395642 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394212    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-lib-modules\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.395642 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-run-openvswitch\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.395642 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394236    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-multus-socket-dir-parent\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.396690 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394252    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-socket-dir\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:18:55.396690 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-run-ovn\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.396690 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394282    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-etc-openvswitch\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.396690 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394299    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-cni-netd\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.396690 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394273    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2aa25e13-aaa4-4795-a221-2056b76e8c8f-host-slash\") pod \"iptables-alerter-9x2kw\" (UID: \"2aa25e13-aaa4-4795-a221-2056b76e8c8f\") " pod="openshift-network-operator/iptables-alerter-9x2kw"
Oct 31 03:18:55.396690 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j92zc\" (UniqueName: \"kubernetes.io/projected/65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce-kube-api-access-j92zc\") pod \"node-resolver-j86qh\" (UID: \"65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce\") " pod="openshift-dns/node-resolver-j86qh"
Oct 31 03:18:55.396690 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394320    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-run-systemd\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.396690 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394340    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-run-ovn\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.396690 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-kubelet-config\") pod \"global-pull-secret-syncer-7nb9d\" (UID: \"8d251d9b-c5e7-46e7-b5ab-524b8171d83b\") " pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:18:55.396690 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394353    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-socket-dir\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:18:55.396690 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394370    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-sys\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.396690 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394370    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-cni-netd\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.396690 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-tmp\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.396690 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-var-lib-cni-bin\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.396690 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394412    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-kubelet-config\") pod \"global-pull-secret-syncer-7nb9d\" (UID: \"8d251d9b-c5e7-46e7-b5ab-524b8171d83b\") " pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:18:55.396690 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394430    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-sys\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.396690 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394440    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-var-lib-kubelet\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.396690 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394456    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-var-lib-cni-bin\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.397620 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-run-netns\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.397620 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394490    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-var-lib-kubelet\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.397620 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394495    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-node-log\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.397620 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394517    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-run-netns\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.397620 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394527    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-node-log\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.397620 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-sys-fs\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:18:55.397620 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394551    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-sys-fs\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:18:55.397620 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-cnibin\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:18:55.397620 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394599    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-modprobe-d\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.397620 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394617    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-cnibin\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:18:55.397620 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394623    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-system-cni-dir\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.397620 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394603    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-dbus\") pod \"global-pull-secret-syncer-7nb9d\" (UID: \"8d251d9b-c5e7-46e7-b5ab-524b8171d83b\") " pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:18:55.397620 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-hostroot\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.397620 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d72ff8d-4480-4266-ab8b-a225f4245836-ovnkube-script-lib\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.397620 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394676    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-modprobe-d\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.397620 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394677    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-system-cni-dir\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.397620 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394715    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-hostroot\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.397620 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394733    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce-hosts-file\") pod \"node-resolver-j86qh\" (UID: \"65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce\") " pod="openshift-dns/node-resolver-j86qh"
Oct 31 03:18:55.398529 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394739    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-system-cni-dir\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:18:55.398529 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394758    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-cni-binary-copy\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:18:55.398529 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394786    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-system-cni-dir\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:18:55.398529 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-sysconfig\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.398529 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394837    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-sysctl-conf\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.398529 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2aa25e13-aaa4-4795-a221-2056b76e8c8f-iptables-alerter-script\") pod \"iptables-alerter-9x2kw\" (UID: \"2aa25e13-aaa4-4795-a221-2056b76e8c8f\") " pod="openshift-network-operator/iptables-alerter-9x2kw"
Oct 31 03:18:55.398529 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394877    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-sysconfig\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.398529 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d72ff8d-4480-4266-ab8b-a225f4245836-ovn-node-metrics-cert\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.398529 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394911    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjkmq\" (UniqueName: \"kubernetes.io/projected/7d72ff8d-4480-4266-ab8b-a225f4245836-kube-api-access-jjkmq\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.398529 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394936    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:18:55.398529 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394960    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-sysctl-d\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.398529 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394970    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-sysctl-conf\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.398529 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.394983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d72ff8d-4480-4266-ab8b-a225f4245836-ovnkube-config\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.398529 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.395010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zzzr\" (UniqueName: \"kubernetes.io/projected/74face88-7a89-4b8f-9e8b-adea29761ed8-kube-api-access-5zzzr\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.398529 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.395033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp4mc\" (UniqueName: \"kubernetes.io/projected/ebb8e08d-8243-492f-aef5-d3b6590d9f47-kube-api-access-cp4mc\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:18:55.398529 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.395202    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d72ff8d-4480-4266-ab8b-a225f4245836-ovnkube-script-lib\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.398529 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.395213    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-cni-binary-copy\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:18:55.399048 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.395316    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-sysctl-d\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.399048 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.395672    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d72ff8d-4480-4266-ab8b-a225f4245836-ovnkube-config\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.399048 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.395707    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:18:55.399048 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.395826    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2aa25e13-aaa4-4795-a221-2056b76e8c8f-iptables-alerter-script\") pod \"iptables-alerter-9x2kw\" (UID: \"2aa25e13-aaa4-4795-a221-2056b76e8c8f\") " pod="openshift-network-operator/iptables-alerter-9x2kw"
Oct 31 03:18:55.399048 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.396749    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ff0da67d-c75e-4487-ba6a-b85e04a496d9-agent-certs\") pod \"konnectivity-agent-kqgxt\" (UID: \"ff0da67d-c75e-4487-ba6a-b85e04a496d9\") " pod="kube-system/konnectivity-agent-kqgxt"
Oct 31 03:18:55.399048 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.396865    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-tuned\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.399048 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.397833    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d72ff8d-4480-4266-ab8b-a225f4245836-ovn-node-metrics-cert\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.399048 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.398023    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-tmp\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.452843 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:55.452811    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:18:55.452843 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:55.452842    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:18:55.453166 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:55.452857    2577 projected.go:196] Error preparing data for projected volume kube-api-access-p5jxx for pod openshift-network-diagnostics/network-check-target-rtp6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:18:55.453166 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:55.452971    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8-kube-api-access-p5jxx podName:bf1633f9-bdb5-4a26-bd80-a83a4b5587e8 nodeName:}" failed. No retries permitted until 2026-10-31 03:18:55.952950366 +0000 UTC m=+2.364228895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p5jxx" (UniqueName: "kubernetes.io/projected/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8-kube-api-access-p5jxx") pod "network-check-target-rtp6w" (UID: "bf1633f9-bdb5-4a26-bd80-a83a4b5587e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:18:55.454916 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.454879    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tpl7\" (UniqueName: \"kubernetes.io/projected/2aa25e13-aaa4-4795-a221-2056b76e8c8f-kube-api-access-6tpl7\") pod \"iptables-alerter-9x2kw\" (UID: \"2aa25e13-aaa4-4795-a221-2056b76e8c8f\") " pod="openshift-network-operator/iptables-alerter-9x2kw"
Oct 31 03:18:55.455616 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.455593    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hjs5\" (UniqueName: \"kubernetes.io/projected/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-kube-api-access-2hjs5\") pod \"network-metrics-daemon-bx98q\" (UID: \"45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9\") " pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:18:55.457161 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.457141    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j92zc\" (UniqueName: \"kubernetes.io/projected/65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce-kube-api-access-j92zc\") pod \"node-resolver-j86qh\" (UID: \"65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce\") " pod="openshift-dns/node-resolver-j86qh"
Oct 31 03:18:55.460782 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.460723    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8ppp\" (UniqueName: \"kubernetes.io/projected/8da6593d-d284-438d-96ee-ec0d9ddf1124-kube-api-access-s8ppp\") pod \"node-ca-2kzdk\" (UID: \"8da6593d-d284-438d-96ee-ec0d9ddf1124\") " pod="openshift-image-registry/node-ca-2kzdk"
Oct 31 03:18:55.461290 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.461262    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4wv2\" (UniqueName: \"kubernetes.io/projected/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-kube-api-access-n4wv2\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:55.462065 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.462016    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zzzr\" (UniqueName: \"kubernetes.io/projected/74face88-7a89-4b8f-9e8b-adea29761ed8-kube-api-access-5zzzr\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:55.462172 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.462142    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjkmq\" (UniqueName: \"kubernetes.io/projected/7d72ff8d-4480-4266-ab8b-a225f4245836-kube-api-access-jjkmq\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.462172 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.462161    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2jf8\" (UniqueName: \"kubernetes.io/projected/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-kube-api-access-b2jf8\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:18:55.463459 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.463437    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp4mc\" (UniqueName: \"kubernetes.io/projected/ebb8e08d-8243-492f-aef5-d3b6590d9f47-kube-api-access-cp4mc\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:18:55.561281 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.561250    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:18:55.574085 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:55.574041    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2075c7dc615e009d2cc914067d053547.slice/crio-6b306c03ddb8748a6e567db9c88ae13d9884f8f0e5b71fcadd5acc1bd96a1623 WatchSource:0}: Error finding container 6b306c03ddb8748a6e567db9c88ae13d9884f8f0e5b71fcadd5acc1bd96a1623: Status 404 returned error can't find the container with id 6b306c03ddb8748a6e567db9c88ae13d9884f8f0e5b71fcadd5acc1bd96a1623
Oct 31 03:18:55.574678 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:55.574651    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21c869370d277ce1b45bad9863021dc.slice/crio-1ba34688772781b5f2a9085ffdcd167ec4f814dbc44c6f390f2e71d567c903b5 WatchSource:0}: Error finding container 1ba34688772781b5f2a9085ffdcd167ec4f814dbc44c6f390f2e71d567c903b5: Status 404 returned error can't find the container with id 1ba34688772781b5f2a9085ffdcd167ec4f814dbc44c6f390f2e71d567c903b5
Oct 31 03:18:55.575310 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:55.575284    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d72ff8d_4480_4266_ab8b_a225f4245836.slice/crio-d51312290a9c7031fe0fa5522d6e9e874f84f23859cda8a6a5e7dc0bff8e8fbc WatchSource:0}: Error finding container d51312290a9c7031fe0fa5522d6e9e874f84f23859cda8a6a5e7dc0bff8e8fbc: Status 404 returned error can't find the container with id d51312290a9c7031fe0fa5522d6e9e874f84f23859cda8a6a5e7dc0bff8e8fbc
Oct 31 03:18:55.580133 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.580114    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:18:55.590283 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:55.590264    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:18:56.642373 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:56.642352    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb8e08d_8243_492f_aef5_d3b6590d9f47.slice/crio-c65a43209eec3ae1f9aae6e32e4f0e7e8b6b7ca011b7a2f5b7964969ab0dd3a3 WatchSource:0}: Error finding container c65a43209eec3ae1f9aae6e32e4f0e7e8b6b7ca011b7a2f5b7964969ab0dd3a3: Status 404 returned error can't find the container with id c65a43209eec3ae1f9aae6e32e4f0e7e8b6b7ca011b7a2f5b7964969ab0dd3a3
Oct 31 03:18:56.647596 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:56.647578    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2kzdk"
Oct 31 03:18:56.655951 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:56.655926    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da6593d_d284_438d_96ee_ec0d9ddf1124.slice/crio-bba2411ce2baf8a44cbc6b4ccfbeb6de781689e24f146e060d96d64fdb1d5c18 WatchSource:0}: Error finding container bba2411ce2baf8a44cbc6b4ccfbeb6de781689e24f146e060d96d64fdb1d5c18: Status 404 returned error can't find the container with id bba2411ce2baf8a44cbc6b4ccfbeb6de781689e24f146e060d96d64fdb1d5c18
Oct 31 03:18:56.667726 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:56.667695    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kqgxt"
Oct 31 03:18:56.677161 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:56.677137    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff0da67d_c75e_4487_ba6a_b85e04a496d9.slice/crio-ef4e18b14b5e5a470e8bd53b4789e825597d4221cfac7207bc810d3fbce498a8 WatchSource:0}: Error finding container ef4e18b14b5e5a470e8bd53b4789e825597d4221cfac7207bc810d3fbce498a8: Status 404 returned error can't find the container with id ef4e18b14b5e5a470e8bd53b4789e825597d4221cfac7207bc810d3fbce498a8
Oct 31 03:18:56.681358 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:56.681342    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j86qh"
Oct 31 03:18:56.689234 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:56.689208    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c7fc32_ca2a_43f3_a2e6_fe23ee3480ce.slice/crio-ec067d0b6556a679cfccec988c00f2d6cc264e41aaae4cc591b7d6c03c5b8512 WatchSource:0}: Error finding container ec067d0b6556a679cfccec988c00f2d6cc264e41aaae4cc591b7d6c03c5b8512: Status 404 returned error can't find the container with id ec067d0b6556a679cfccec988c00f2d6cc264e41aaae4cc591b7d6c03c5b8512
Oct 31 03:18:56.699294 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:56.699269    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:18:56.702425 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:56.702405    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9x2kw"
Oct 31 03:18:56.709013 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:56.708989    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb6233e_08f7_4a15_b273_b705bf9c2ac7.slice/crio-3ba1e5f242ab5ff991d731d0b55cbe7bc647054d69432e3870be8b507e1c5df1 WatchSource:0}: Error finding container 3ba1e5f242ab5ff991d731d0b55cbe7bc647054d69432e3870be8b507e1c5df1: Status 404 returned error can't find the container with id 3ba1e5f242ab5ff991d731d0b55cbe7bc647054d69432e3870be8b507e1c5df1
Oct 31 03:18:56.711723 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:56.711696    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa25e13_aaa4_4795_a221_2056b76e8c8f.slice/crio-085b4d9646162e9856a356ee4e254e4a9a0aeca688bbdbc97d29c988a7c603cb WatchSource:0}: Error finding container 085b4d9646162e9856a356ee4e254e4a9a0aeca688bbdbc97d29c988a7c603cb: Status 404 returned error can't find the container with id 085b4d9646162e9856a356ee4e254e4a9a0aeca688bbdbc97d29c988a7c603cb
Oct 31 03:18:56.718182 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:56.718164    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:18:56.724002 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:56.723985    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vxkwg"
Oct 31 03:18:56.727457 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:56.727434    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95bd2ef0_32a9_4150_abee_d2a9e03cc62c.slice/crio-412623e31a6e2c7bc98ac1ae7b61a77c0bbcc67c1681898d768330db7519e0f2 WatchSource:0}: Error finding container 412623e31a6e2c7bc98ac1ae7b61a77c0bbcc67c1681898d768330db7519e0f2: Status 404 returned error can't find the container with id 412623e31a6e2c7bc98ac1ae7b61a77c0bbcc67c1681898d768330db7519e0f2
Oct 31 03:18:56.732210 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:18:56.732191    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74face88_7a89_4b8f_9e8b_adea29761ed8.slice/crio-4195bb655ad65a47f514d13d6332248916eb5bb68e854977c1030d712ad41c51 WatchSource:0}: Error finding container 4195bb655ad65a47f514d13d6332248916eb5bb68e854977c1030d712ad41c51: Status 404 returned error can't find the container with id 4195bb655ad65a47f514d13d6332248916eb5bb68e854977c1030d712ad41c51
Oct 31 03:18:56.733943 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:56.733923    2577 kuberuntime_manager.go:1707] "Unhandled Error" err="container kube-multus start failed in pod multus-vxkwg_openshift-multus(74face88-7a89-4b8f-9e8b-adea29761ed8): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Oct 31 03:18:56.734017 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:56.733971    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-multus\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-multus/multus-vxkwg" podUID="74face88-7a89-4b8f-9e8b-adea29761ed8"
Oct 31 03:18:56.933090 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:56.933013    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:18:56.940628 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:56.940603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-original-pull-secret\") pod \"global-pull-secret-syncer-7nb9d\" (UID: \"8d251d9b-c5e7-46e7-b5ab-524b8171d83b\") " pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:18:56.940730 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:56.940651    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs\") pod \"network-metrics-daemon-bx98q\" (UID: \"45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9\") " pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:18:56.940730 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:56.940695    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:18:56.940730 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:56.940730    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:18:56.940840 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:56.940748    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-original-pull-secret podName:8d251d9b-c5e7-46e7-b5ab-524b8171d83b nodeName:}" failed. No retries permitted until 2026-10-31 03:18:57.940734716 +0000 UTC m=+3.309222048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-original-pull-secret") pod "global-pull-secret-syncer-7nb9d" (UID: "8d251d9b-c5e7-46e7-b5ab-524b8171d83b") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:18:56.940840 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:56.940768    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs podName:45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9 nodeName:}" failed. No retries permitted until 2026-10-31 03:18:57.94075743 +0000 UTC m=+3.309244749 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs") pod "network-metrics-daemon-bx98q" (UID: "45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:18:57.041242 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:57.041209    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5jxx\" (UniqueName: \"kubernetes.io/projected/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8-kube-api-access-p5jxx\") pod \"network-check-target-rtp6w\" (UID: \"bf1633f9-bdb5-4a26-bd80-a83a4b5587e8\") " pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:18:57.041847 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:57.041336    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:18:57.041847 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:57.041350    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:18:57.041847 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:57.041363    2577 projected.go:196] Error preparing data for projected volume kube-api-access-p5jxx for pod openshift-network-diagnostics/network-check-target-rtp6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:18:57.041847 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:57.041409    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8-kube-api-access-p5jxx podName:bf1633f9-bdb5-4a26-bd80-a83a4b5587e8 nodeName:}" failed. No retries permitted until 2026-10-31 03:18:58.041396951 +0000 UTC m=+3.409884269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p5jxx" (UniqueName: "kubernetes.io/projected/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8-kube-api-access-p5jxx") pod "network-check-target-rtp6w" (UID: "bf1633f9-bdb5-4a26-bd80-a83a4b5587e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:18:57.117320 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:57.117260    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkj57" event={"ID":"bcb6233e-08f7-4a15-b273-b705bf9c2ac7","Type":"ContainerStarted","Data":"3ba1e5f242ab5ff991d731d0b55cbe7bc647054d69432e3870be8b507e1c5df1"}
Oct 31 03:18:57.118259 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:57.118241    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kqgxt" event={"ID":"ff0da67d-c75e-4487-ba6a-b85e04a496d9","Type":"ContainerStarted","Data":"ef4e18b14b5e5a470e8bd53b4789e825597d4221cfac7207bc810d3fbce498a8"}
Oct 31 03:18:57.119149 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:57.119130    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2kzdk" event={"ID":"8da6593d-d284-438d-96ee-ec0d9ddf1124","Type":"ContainerStarted","Data":"bba2411ce2baf8a44cbc6b4ccfbeb6de781689e24f146e060d96d64fdb1d5c18"}
Oct 31 03:18:57.120128 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:57.120108    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp" event={"ID":"ebb8e08d-8243-492f-aef5-d3b6590d9f47","Type":"ContainerStarted","Data":"c65a43209eec3ae1f9aae6e32e4f0e7e8b6b7ca011b7a2f5b7964969ab0dd3a3"}
Oct 31 03:18:57.120987 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:57.120968    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal" event={"ID":"d21c869370d277ce1b45bad9863021dc","Type":"ContainerStarted","Data":"1ba34688772781b5f2a9085ffdcd167ec4f814dbc44c6f390f2e71d567c903b5"}
Oct 31 03:18:57.121926 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:57.121905    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vxkwg" event={"ID":"74face88-7a89-4b8f-9e8b-adea29761ed8","Type":"ContainerStarted","Data":"4195bb655ad65a47f514d13d6332248916eb5bb68e854977c1030d712ad41c51"}
Oct 31 03:18:57.122870 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:57.122849    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l494z" event={"ID":"95bd2ef0-32a9-4150-abee-d2a9e03cc62c","Type":"ContainerStarted","Data":"412623e31a6e2c7bc98ac1ae7b61a77c0bbcc67c1681898d768330db7519e0f2"}
Oct 31 03:18:57.123450 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:57.123429    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-multus\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b14b49cabfdc258931fa03823e067322606fe50d514622d8c4f0efaadb17d1f0\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-multus/multus-vxkwg" podUID="74face88-7a89-4b8f-9e8b-adea29761ed8"
Oct 31 03:18:57.123812 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:57.123798    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j86qh" event={"ID":"65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce","Type":"ContainerStarted","Data":"ec067d0b6556a679cfccec988c00f2d6cc264e41aaae4cc591b7d6c03c5b8512"}
Oct 31 03:18:57.124770 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:57.124751    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4" event={"ID":"7d72ff8d-4480-4266-ab8b-a225f4245836","Type":"ContainerStarted","Data":"d51312290a9c7031fe0fa5522d6e9e874f84f23859cda8a6a5e7dc0bff8e8fbc"}
Oct 31 03:18:57.125627 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:57.125609    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal" event={"ID":"2075c7dc615e009d2cc914067d053547","Type":"ContainerStarted","Data":"6b306c03ddb8748a6e567db9c88ae13d9884f8f0e5b71fcadd5acc1bd96a1623"}
Oct 31 03:18:57.126525 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:57.126508    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9x2kw" event={"ID":"2aa25e13-aaa4-4795-a221-2056b76e8c8f","Type":"ContainerStarted","Data":"085b4d9646162e9856a356ee4e254e4a9a0aeca688bbdbc97d29c988a7c603cb"}
Oct 31 03:18:57.126639 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:57.126627    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:18:57.417161 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:57.417063    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:13:56 +0000 UTC" deadline="2028-04-21 10:02:55.285331528 +0000 UTC"
Oct 31 03:18:57.417161 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:57.417104    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12918h43m57.868230132s"
Oct 31 03:18:57.452370 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:57.452331    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:18:57.947686 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:57.947652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs\") pod \"network-metrics-daemon-bx98q\" (UID: \"45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9\") " pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:18:57.947908 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:57.947732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-original-pull-secret\") pod \"global-pull-secret-syncer-7nb9d\" (UID: \"8d251d9b-c5e7-46e7-b5ab-524b8171d83b\") " pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:18:57.947994 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:57.947977    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:18:57.948051 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:57.948043    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs podName:45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9 nodeName:}" failed. No retries permitted until 2026-10-31 03:18:59.948025478 +0000 UTC m=+5.316512811 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs") pod "network-metrics-daemon-bx98q" (UID: "45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:18:57.948093 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:57.948086    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:18:57.948132 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:57.948122    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-original-pull-secret podName:8d251d9b-c5e7-46e7-b5ab-524b8171d83b nodeName:}" failed. No retries permitted until 2026-10-31 03:18:59.948112128 +0000 UTC m=+5.316599460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-original-pull-secret") pod "global-pull-secret-syncer-7nb9d" (UID: "8d251d9b-c5e7-46e7-b5ab-524b8171d83b") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:18:58.048481 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:58.048446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5jxx\" (UniqueName: \"kubernetes.io/projected/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8-kube-api-access-p5jxx\") pod \"network-check-target-rtp6w\" (UID: \"bf1633f9-bdb5-4a26-bd80-a83a4b5587e8\") " pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:18:58.048985 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:58.048647    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:18:58.048985 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:58.048670    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:18:58.048985 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:58.048683    2577 projected.go:196] Error preparing data for projected volume kube-api-access-p5jxx for pod openshift-network-diagnostics/network-check-target-rtp6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:18:58.048985 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:58.048738    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8-kube-api-access-p5jxx podName:bf1633f9-bdb5-4a26-bd80-a83a4b5587e8 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:00.048720205 +0000 UTC m=+5.417207526 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-p5jxx" (UniqueName: "kubernetes.io/projected/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8-kube-api-access-p5jxx") pod "network-check-target-rtp6w" (UID: "bf1633f9-bdb5-4a26-bd80-a83a4b5587e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:18:58.113450 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:58.113414    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:18:58.113636 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:58.113559    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nb9d" podUID="8d251d9b-c5e7-46e7-b5ab-524b8171d83b"
Oct 31 03:18:58.113692 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:58.113680    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:18:58.113767 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:58.113748    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtp6w" podUID="bf1633f9-bdb5-4a26-bd80-a83a4b5587e8"
Oct 31 03:18:58.113874 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:58.113818    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:18:58.113960 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:58.113909    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx98q" podUID="45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9"
Oct 31 03:18:58.418276 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:58.418182    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:13:56 +0000 UTC" deadline="2028-07-09 12:56:27.871651884 +0000 UTC"
Oct 31 03:18:58.418276 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:58.418229    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14817h37m29.453428356s"
Oct 31 03:18:59.967207 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:59.966804    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs\") pod \"network-metrics-daemon-bx98q\" (UID: \"45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9\") " pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:18:59.967207 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:18:59.966860    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-original-pull-secret\") pod \"global-pull-secret-syncer-7nb9d\" (UID: \"8d251d9b-c5e7-46e7-b5ab-524b8171d83b\") " pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:18:59.967207 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:59.967007    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:18:59.967207 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:59.967079    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-original-pull-secret podName:8d251d9b-c5e7-46e7-b5ab-524b8171d83b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:03.967061667 +0000 UTC m=+9.335549000 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-original-pull-secret") pod "global-pull-secret-syncer-7nb9d" (UID: "8d251d9b-c5e7-46e7-b5ab-524b8171d83b") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:18:59.967207 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:59.967145    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:18:59.967207 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:18:59.967178    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs podName:45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:03.967167562 +0000 UTC m=+9.335654895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs") pod "network-metrics-daemon-bx98q" (UID: "45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:19:00.069357 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:00.069014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5jxx\" (UniqueName: \"kubernetes.io/projected/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8-kube-api-access-p5jxx\") pod \"network-check-target-rtp6w\" (UID: \"bf1633f9-bdb5-4a26-bd80-a83a4b5587e8\") " pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:19:00.069357 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:00.069213    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:19:00.069357 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:00.069235    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:19:00.069357 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:00.069248    2577 projected.go:196] Error preparing data for projected volume kube-api-access-p5jxx for pod openshift-network-diagnostics/network-check-target-rtp6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:19:00.069357 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:00.069305    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8-kube-api-access-p5jxx podName:bf1633f9-bdb5-4a26-bd80-a83a4b5587e8 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:04.069286746 +0000 UTC m=+9.437774087 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-p5jxx" (UniqueName: "kubernetes.io/projected/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8-kube-api-access-p5jxx") pod "network-check-target-rtp6w" (UID: "bf1633f9-bdb5-4a26-bd80-a83a4b5587e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:19:00.114066 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:00.113590    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:19:00.114066 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:00.113722    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtp6w" podUID="bf1633f9-bdb5-4a26-bd80-a83a4b5587e8"
Oct 31 03:19:00.114066 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:00.113794    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:19:00.114066 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:00.113904    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx98q" podUID="45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9"
Oct 31 03:19:00.114066 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:00.113953    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:00.114066 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:00.114024    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nb9d" podUID="8d251d9b-c5e7-46e7-b5ab-524b8171d83b"
Oct 31 03:19:01.155540 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:01.155481    2577 generic.go:356] "Generic (PLEG): container finished" podID="bcb6233e-08f7-4a15-b273-b705bf9c2ac7" containerID="005714943d9331bce0ebde8ebf14dc484731ba562309490cb77c206e989413a4" exitCode=0
Oct 31 03:19:01.155540 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:01.155541    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkj57" event={"ID":"bcb6233e-08f7-4a15-b273-b705bf9c2ac7","Type":"ContainerDied","Data":"005714943d9331bce0ebde8ebf14dc484731ba562309490cb77c206e989413a4"}
Oct 31 03:19:02.114515 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:02.114064    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:02.114515 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:02.114206    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nb9d" podUID="8d251d9b-c5e7-46e7-b5ab-524b8171d83b"
Oct 31 03:19:02.114515 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:02.114285    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:19:02.114515 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:02.114349    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtp6w" podUID="bf1633f9-bdb5-4a26-bd80-a83a4b5587e8"
Oct 31 03:19:02.114515 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:02.114400    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:19:02.114515 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:02.114471    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx98q" podUID="45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9"
Oct 31 03:19:02.165201 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:02.165167    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kqgxt" event={"ID":"ff0da67d-c75e-4487-ba6a-b85e04a496d9","Type":"ContainerStarted","Data":"26a7c57d096ccee1d27289d6f4e6ebeb00ba00c0c28c417a21c9f5a1fc001c01"}
Oct 31 03:19:02.168902 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:02.168847    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal" event={"ID":"d21c869370d277ce1b45bad9863021dc","Type":"ContainerStarted","Data":"9ae1bca4c6062418d01e42e9feccd0a77c0108459338f83cb0271395dc707349"}
Oct 31 03:19:02.174364 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:02.174332    2577 generic.go:356] "Generic (PLEG): container finished" podID="2075c7dc615e009d2cc914067d053547" containerID="88711d43e6441034cbc31cc514ab04d40f74f4e6902efd78b50795b3e7ea72f5" exitCode=0
Oct 31 03:19:02.174491 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:02.174382    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal" event={"ID":"2075c7dc615e009d2cc914067d053547","Type":"ContainerDied","Data":"88711d43e6441034cbc31cc514ab04d40f74f4e6902efd78b50795b3e7ea72f5"}
Oct 31 03:19:02.397661 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:02.396113    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kqgxt" podStartSLOduration=2.978642143 podStartE2EDuration="7.396093783s" totalImagesPullingTime="4.41745164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:02.281421055 +0000 UTC m=+7.649908398" watchObservedRunningTime="2026-10-31 03:19:02.396093783 +0000 UTC m=+7.764581128"
Oct 31 03:19:02.513177 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:02.513122    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal" podStartSLOduration=7.513105314 podStartE2EDuration="7.513105314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:02.397416329 +0000 UTC m=+7.765903661" watchObservedRunningTime="2026-10-31 03:19:02.513105314 +0000 UTC m=+7.881592655"
Oct 31 03:19:03.181937 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:03.181859    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal" event={"ID":"2075c7dc615e009d2cc914067d053547","Type":"ContainerStarted","Data":"cedb4e29fc2f98c29b526d5519c012c78bd0cc60d2e1393ff7d12ff7e4443f83"}
Oct 31 03:19:03.184547 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:03.184521    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2kzdk" event={"ID":"8da6593d-d284-438d-96ee-ec0d9ddf1124","Type":"ContainerStarted","Data":"deed9c2b1ae44b8d526ecde4ebf3be61f51e12d858bab29fa80cc376d91a6f35"}
Oct 31 03:19:03.186839 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:03.186815    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp" event={"ID":"ebb8e08d-8243-492f-aef5-d3b6590d9f47","Type":"ContainerStarted","Data":"763273f3010ddd0c4accc991ecf71fc25794e1e1f8734d571886ad509da1cc9b"}
Oct 31 03:19:03.371910 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:03.371853    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kqgxt"
Oct 31 03:19:03.372809 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:03.372786    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kqgxt"
Oct 31 03:19:03.414186 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:03.413217    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2kzdk" podStartSLOduration=2.823921123 podStartE2EDuration="8.41320014s" totalImagesPullingTime="5.589279017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:03.412830052 +0000 UTC m=+8.781317395" watchObservedRunningTime="2026-10-31 03:19:03.41320014 +0000 UTC m=+8.781687482"
Oct 31 03:19:03.414186 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:03.413968    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal" podStartSLOduration=8.4139585 podStartE2EDuration="8.4139585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:03.298630645 +0000 UTC m=+8.667117988" watchObservedRunningTime="2026-10-31 03:19:03.4139585 +0000 UTC m=+8.782445844"
Oct 31 03:19:04.006702 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:04.006619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs\") pod \"network-metrics-daemon-bx98q\" (UID: \"45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9\") " pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:19:04.006702 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:04.006679    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-original-pull-secret\") pod \"global-pull-secret-syncer-7nb9d\" (UID: \"8d251d9b-c5e7-46e7-b5ab-524b8171d83b\") " pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:04.006924 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:04.006803    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:19:04.006924 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:04.006862    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-original-pull-secret podName:8d251d9b-c5e7-46e7-b5ab-524b8171d83b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:12.006843549 +0000 UTC m=+17.375330873 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-original-pull-secret") pod "global-pull-secret-syncer-7nb9d" (UID: "8d251d9b-c5e7-46e7-b5ab-524b8171d83b") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:19:04.007045 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:04.006958    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:19:04.007045 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:04.006999    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs podName:45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:12.006985389 +0000 UTC m=+17.375472709 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs") pod "network-metrics-daemon-bx98q" (UID: "45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:19:04.107491 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:04.107455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5jxx\" (UniqueName: \"kubernetes.io/projected/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8-kube-api-access-p5jxx\") pod \"network-check-target-rtp6w\" (UID: \"bf1633f9-bdb5-4a26-bd80-a83a4b5587e8\") " pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:19:04.107670 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:04.107627    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:19:04.107670 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:04.107667    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:19:04.107769 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:04.107679    2577 projected.go:196] Error preparing data for projected volume kube-api-access-p5jxx for pod openshift-network-diagnostics/network-check-target-rtp6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:19:04.107769 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:04.107745    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8-kube-api-access-p5jxx podName:bf1633f9-bdb5-4a26-bd80-a83a4b5587e8 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:12.107726893 +0000 UTC m=+17.476214225 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-p5jxx" (UniqueName: "kubernetes.io/projected/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8-kube-api-access-p5jxx") pod "network-check-target-rtp6w" (UID: "bf1633f9-bdb5-4a26-bd80-a83a4b5587e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:19:04.114103 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:04.113620    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:19:04.114103 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:04.113757    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx98q" podUID="45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9"
Oct 31 03:19:04.114103 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:04.113832    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:04.114103 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:04.113934    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nb9d" podUID="8d251d9b-c5e7-46e7-b5ab-524b8171d83b"
Oct 31 03:19:04.114103 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:04.113996    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:19:04.114103 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:04.114064    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtp6w" podUID="bf1633f9-bdb5-4a26-bd80-a83a4b5587e8"
Oct 31 03:19:04.934769 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:04.934736    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 03:19:05.098534 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:05.098431    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T03:19:04.934757816Z","UUID":"0a41e6ad-36e8-4725-a025-b0e93d1bcbd3","Handler":null,"Name":"","Endpoint":""}
Oct 31 03:19:05.100081 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:05.100054    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 03:19:05.100225 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:05.100089    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 03:19:05.193845 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:05.193726    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp" event={"ID":"ebb8e08d-8243-492f-aef5-d3b6590d9f47","Type":"ContainerStarted","Data":"6db211896f47465bcc74c6cf4eacf28d7aff88b1942b16ff17cc16b6c388fd98"}
Oct 31 03:19:05.195382 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:05.195355    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 03:19:05.195785 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:05.195762    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j86qh" event={"ID":"65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce","Type":"ContainerStarted","Data":"a6e0fe95e6978684bf5d84d9219ebc6a88cfdeefc0d2be651bafca8916f3b821"}
Oct 31 03:19:05.312414 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:05.312369    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-j86qh" podStartSLOduration=2.391326493 podStartE2EDuration="10.312353228s" totalImagesPullingTime="7.921026735s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:05.311463603 +0000 UTC m=+10.679950942" watchObservedRunningTime="2026-10-31 03:19:05.312353228 +0000 UTC m=+10.680840569"
Oct 31 03:19:06.114275 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:06.113926    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:19:06.114275 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:06.113974    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:19:06.114275 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:06.114057    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtp6w" podUID="bf1633f9-bdb5-4a26-bd80-a83a4b5587e8"
Oct 31 03:19:06.114275 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:06.113934    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:06.114275 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:06.114172    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx98q" podUID="45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9"
Oct 31 03:19:06.114275 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:06.114245    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nb9d" podUID="8d251d9b-c5e7-46e7-b5ab-524b8171d83b"
Oct 31 03:19:06.199563 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:06.199529    2577 generic.go:356] "Generic (PLEG): container finished" podID="bcb6233e-08f7-4a15-b273-b705bf9c2ac7" containerID="3dd64bb5386f6ecef726aae51ade85cfef7dca5a50665271b6e3e71830b0cdb1" exitCode=0
Oct 31 03:19:06.199993 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:06.199963    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkj57" event={"ID":"bcb6233e-08f7-4a15-b273-b705bf9c2ac7","Type":"ContainerDied","Data":"3dd64bb5386f6ecef726aae51ade85cfef7dca5a50665271b6e3e71830b0cdb1"}
Oct 31 03:19:07.204065 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:07.203811    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l494z" event={"ID":"95bd2ef0-32a9-4150-abee-d2a9e03cc62c","Type":"ContainerStarted","Data":"cd3163301b9df47e3eb6439c0a29323244b9a30482cfacb80087a679c5e998c5"}
Oct 31 03:19:07.205475 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:07.205419    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9x2kw" event={"ID":"2aa25e13-aaa4-4795-a221-2056b76e8c8f","Type":"ContainerStarted","Data":"4e67540f6a6105ab1d6148bfcadb0f78e32921c47024925c332aa0ba16cd415a"}
Oct 31 03:19:07.208307 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:07.208279    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp" event={"ID":"ebb8e08d-8243-492f-aef5-d3b6590d9f47","Type":"ContainerStarted","Data":"1f8a5a0d58262c2ba9d4a0f0de10e8e50cb926bccd1bb5695f18a87cd58e0f4f"}
Oct 31 03:19:07.437091 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:07.437030    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9x2kw" podStartSLOduration=4.350769445 podStartE2EDuration="12.43701572s" totalImagesPullingTime="8.086246275s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:07.436796026 +0000 UTC m=+12.805283366" watchObservedRunningTime="2026-10-31 03:19:07.43701572 +0000 UTC m=+12.805503059"
Oct 31 03:19:07.437251 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:07.437163    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-l494z" podStartSLOduration=2.542512997 podStartE2EDuration="12.437155157s" totalImagesPullingTime="9.89464216s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:07.322066368 +0000 UTC m=+12.690553708" watchObservedRunningTime="2026-10-31 03:19:07.437155157 +0000 UTC m=+12.805642497"
Oct 31 03:19:07.558035 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:07.557935    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp" podStartSLOduration=3.213751125 podStartE2EDuration="12.557916825s" totalImagesPullingTime="9.3441657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:07.557564041 +0000 UTC m=+12.926051382" watchObservedRunningTime="2026-10-31 03:19:07.557916825 +0000 UTC m=+12.926404172"
Oct 31 03:19:08.113294 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:08.113253    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:19:08.113489 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:08.113337    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:08.113489 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:08.113452    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:19:08.113489 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:08.113465    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtp6w" podUID="bf1633f9-bdb5-4a26-bd80-a83a4b5587e8"
Oct 31 03:19:08.113651 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:08.113585    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx98q" podUID="45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9"
Oct 31 03:19:08.113761 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:08.113677    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nb9d" podUID="8d251d9b-c5e7-46e7-b5ab-524b8171d83b"
Oct 31 03:19:08.212476 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:08.212405    2577 generic.go:356] "Generic (PLEG): container finished" podID="bcb6233e-08f7-4a15-b273-b705bf9c2ac7" containerID="bc234353a20d91ee34a30ffc06fdc8903a0797604e3b25e10fdce212de1fea20" exitCode=0
Oct 31 03:19:08.212476 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:08.212465    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkj57" event={"ID":"bcb6233e-08f7-4a15-b273-b705bf9c2ac7","Type":"ContainerDied","Data":"bc234353a20d91ee34a30ffc06fdc8903a0797604e3b25e10fdce212de1fea20"}
Oct 31 03:19:10.114257 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:10.114185    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:19:10.115000 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:10.114291    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtp6w" podUID="bf1633f9-bdb5-4a26-bd80-a83a4b5587e8"
Oct 31 03:19:10.115000 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:10.114673    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:19:10.115000 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:10.114775    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx98q" podUID="45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9"
Oct 31 03:19:10.115000 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:10.114854    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:10.115000 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:10.114936    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nb9d" podUID="8d251d9b-c5e7-46e7-b5ab-524b8171d83b"
Oct 31 03:19:10.219345 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:10.219309    2577 generic.go:356] "Generic (PLEG): container finished" podID="bcb6233e-08f7-4a15-b273-b705bf9c2ac7" containerID="6fb7c479af725e661f4e4368c7dbdaf1e8885810c6906443a8d958e6a39fe475" exitCode=0
Oct 31 03:19:10.219527 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:10.219361    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkj57" event={"ID":"bcb6233e-08f7-4a15-b273-b705bf9c2ac7","Type":"ContainerDied","Data":"6fb7c479af725e661f4e4368c7dbdaf1e8885810c6906443a8d958e6a39fe475"}
Oct 31 03:19:11.225570 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:11.225249    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 03:19:11.226062 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:11.225807    2577 generic.go:356] "Generic (PLEG): container finished" podID="7d72ff8d-4480-4266-ab8b-a225f4245836" containerID="ccc19a43a88dce5acb62ea67d353bc533a7767d601c403d0d8a4e3a6f6ffd42d" exitCode=1
Oct 31 03:19:11.226062 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:11.225839    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4" event={"ID":"7d72ff8d-4480-4266-ab8b-a225f4245836","Type":"ContainerStarted","Data":"2c591578c888a1f7b378c4a25f39cccbbe6c83e4844e4336c163c98c6a6f3c15"}
Oct 31 03:19:11.226062 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:11.225954    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4" event={"ID":"7d72ff8d-4480-4266-ab8b-a225f4245836","Type":"ContainerStarted","Data":"74bd33c7536dc50585572b8337b1d471e229771fc27a3706787091e219b966ca"}
Oct 31 03:19:11.226062 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:11.226000    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4" event={"ID":"7d72ff8d-4480-4266-ab8b-a225f4245836","Type":"ContainerStarted","Data":"241000ea9b428088771e824596ce1fd1d2fca751a8140e5e1a03dc7827fad390"}
Oct 31 03:19:11.226062 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:11.226027    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4" event={"ID":"7d72ff8d-4480-4266-ab8b-a225f4245836","Type":"ContainerStarted","Data":"a91a6188efce8d2765684c5b3d9188781051d99c62dd7875c58ca091df3dbd4d"}
Oct 31 03:19:11.226062 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:11.226052    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4" event={"ID":"7d72ff8d-4480-4266-ab8b-a225f4245836","Type":"ContainerDied","Data":"ccc19a43a88dce5acb62ea67d353bc533a7767d601c403d0d8a4e3a6f6ffd42d"}
Oct 31 03:19:11.226301 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:11.226089    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4" event={"ID":"7d72ff8d-4480-4266-ab8b-a225f4245836","Type":"ContainerStarted","Data":"c2a3203f831c6537b4dbda791da3e3ecf0db39d25dfb838ae50dfaa04b63e1c0"}
Oct 31 03:19:12.071383 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:12.070614    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs\") pod \"network-metrics-daemon-bx98q\" (UID: \"45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9\") " pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:19:12.071383 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:12.070674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-original-pull-secret\") pod \"global-pull-secret-syncer-7nb9d\" (UID: \"8d251d9b-c5e7-46e7-b5ab-524b8171d83b\") " pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:12.071383 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:12.070843    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:19:12.071383 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:12.070924    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-original-pull-secret podName:8d251d9b-c5e7-46e7-b5ab-524b8171d83b nodeName:}" failed. No retries permitted until 2026-10-31 03:19:28.070904356 +0000 UTC m=+33.439391674 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-original-pull-secret") pod "global-pull-secret-syncer-7nb9d" (UID: "8d251d9b-c5e7-46e7-b5ab-524b8171d83b") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:19:12.071383 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:12.071305    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:19:12.071383 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:12.071352    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs podName:45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:28.071337058 +0000 UTC m=+33.439824381 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs") pod "network-metrics-daemon-bx98q" (UID: "45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:19:12.113520 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:12.113481    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:12.113738 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:12.113489    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:19:12.113738 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:12.113623    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nb9d" podUID="8d251d9b-c5e7-46e7-b5ab-524b8171d83b"
Oct 31 03:19:12.113738 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:12.113501    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:19:12.113738 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:12.113689    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtp6w" podUID="bf1633f9-bdb5-4a26-bd80-a83a4b5587e8"
Oct 31 03:19:12.113930 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:12.113738    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx98q" podUID="45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9"
Oct 31 03:19:12.172083 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:12.172045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5jxx\" (UniqueName: \"kubernetes.io/projected/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8-kube-api-access-p5jxx\") pod \"network-check-target-rtp6w\" (UID: \"bf1633f9-bdb5-4a26-bd80-a83a4b5587e8\") " pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:19:12.172266 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:12.172245    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:19:12.172310 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:12.172273    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:19:12.172310 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:12.172284    2577 projected.go:196] Error preparing data for projected volume kube-api-access-p5jxx for pod openshift-network-diagnostics/network-check-target-rtp6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:19:12.172381 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:12.172344    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8-kube-api-access-p5jxx podName:bf1633f9-bdb5-4a26-bd80-a83a4b5587e8 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:28.172329593 +0000 UTC m=+33.540816915 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-p5jxx" (UniqueName: "kubernetes.io/projected/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8-kube-api-access-p5jxx") pod "network-check-target-rtp6w" (UID: "bf1633f9-bdb5-4a26-bd80-a83a4b5587e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:19:13.232975 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:13.232719    2577 generic.go:356] "Generic (PLEG): container finished" podID="bcb6233e-08f7-4a15-b273-b705bf9c2ac7" containerID="a75232eea5bae4aee8d8e20d8741e2dbd8f852258e4936b5d9b3a1e1b23c0823" exitCode=0
Oct 31 03:19:13.233322 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:13.232797    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkj57" event={"ID":"bcb6233e-08f7-4a15-b273-b705bf9c2ac7","Type":"ContainerDied","Data":"a75232eea5bae4aee8d8e20d8741e2dbd8f852258e4936b5d9b3a1e1b23c0823"}
Oct 31 03:19:14.113543 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:14.113517    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:14.113635 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:14.113554    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:19:14.113635 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:14.113622    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:19:14.113722 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:14.113636    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nb9d" podUID="8d251d9b-c5e7-46e7-b5ab-524b8171d83b"
Oct 31 03:19:14.113722 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:14.113714    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx98q" podUID="45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9"
Oct 31 03:19:14.113802 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:14.113787    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtp6w" podUID="bf1633f9-bdb5-4a26-bd80-a83a4b5587e8"
Oct 31 03:19:14.238407 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:14.238360    2577 generic.go:356] "Generic (PLEG): container finished" podID="bcb6233e-08f7-4a15-b273-b705bf9c2ac7" containerID="ab8d866ba94cbf183fad65bef9d24de7bc51efd5ab783c004d46eba3a58a0740" exitCode=0
Oct 31 03:19:14.239140 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:14.238436    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkj57" event={"ID":"bcb6233e-08f7-4a15-b273-b705bf9c2ac7","Type":"ContainerDied","Data":"ab8d866ba94cbf183fad65bef9d24de7bc51efd5ab783c004d46eba3a58a0740"}
Oct 31 03:19:14.240004 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:14.239979    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vxkwg" event={"ID":"74face88-7a89-4b8f-9e8b-adea29761ed8","Type":"ContainerStarted","Data":"7315601ecfb39ab2cd84440ce67f1b4769350db4c09a0f76a264e1a545d0eadc"}
Oct 31 03:19:14.243417 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:14.243391    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 03:19:14.243800 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:14.243774    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4" event={"ID":"7d72ff8d-4480-4266-ab8b-a225f4245836","Type":"ContainerStarted","Data":"86eb2dbb8e767f0f1f721e22f405fed75dda2ac0c8109801c39e76831e537909"}
Oct 31 03:19:14.514441 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:14.514342    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-vxkwg" podStartSLOduration=2.136570246 podStartE2EDuration="19.514329284s" totalImagesPullingTime="17.377759038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-10-31 03:19:14.514169147 +0000 UTC m=+19.882656485" watchObservedRunningTime="2026-10-31 03:19:14.514329284 +0000 UTC m=+19.882816624"
Oct 31 03:19:15.248289 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:15.248257    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkj57" event={"ID":"bcb6233e-08f7-4a15-b273-b705bf9c2ac7","Type":"ContainerStarted","Data":"50884a606bd0aa30f7462f8a51f53474cd91582126179baf7891795a2acb2232"}
Oct 31 03:19:15.369959 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:15.369917    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lkj57" podStartSLOduration=8.256633884 podStartE2EDuration="20.369903061s" totalImagesPullingTime="12.113269177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:15.36975822 +0000 UTC m=+20.738245560" watchObservedRunningTime="2026-10-31 03:19:15.369903061 +0000 UTC m=+20.738390404"
Oct 31 03:19:16.113635 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:16.113431    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:19:16.113793 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:16.113501    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:19:16.113793 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:16.113727    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx98q" podUID="45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9"
Oct 31 03:19:16.113793 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:16.113774    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtp6w" podUID="bf1633f9-bdb5-4a26-bd80-a83a4b5587e8"
Oct 31 03:19:16.113793 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:16.113509    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:16.113982 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:16.113850    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nb9d" podUID="8d251d9b-c5e7-46e7-b5ab-524b8171d83b"
Oct 31 03:19:16.253693 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:16.253659    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 03:19:16.254282 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:16.254070    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4" event={"ID":"7d72ff8d-4480-4266-ab8b-a225f4245836","Type":"ContainerStarted","Data":"bdbfd0476cb62a25a4b3ff33190f5f145331e51090f688940e951c08d1fab68d"}
Oct 31 03:19:16.256950 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:16.255955    2577 scope.go:122] "RemoveContainer" containerID="ccc19a43a88dce5acb62ea67d353bc533a7767d601c403d0d8a4e3a6f6ffd42d"
Oct 31 03:19:17.259579 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:17.259552    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 03:19:17.260209 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:17.259912    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4" event={"ID":"7d72ff8d-4480-4266-ab8b-a225f4245836","Type":"ContainerStarted","Data":"d9c2ce1fc925c550a603e42afe560a0f72884d633da5b3701d3d3fb6e718c8fe"}
Oct 31 03:19:17.260209 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:17.260191    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:17.260321 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:17.260290    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:17.260361 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:17.260341    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:17.275253 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:17.275223    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:17.279330 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:17.279314    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:17.538699 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:17.538591    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4" podStartSLOduration=8.471333782 podStartE2EDuration="22.538577707s" totalImagesPullingTime="14.067243925s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:17.538451544 +0000 UTC m=+22.906938883" watchObservedRunningTime="2026-10-31 03:19:17.538577707 +0000 UTC m=+22.907065067"
Oct 31 03:19:18.075290 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:18.075246    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7nb9d"]
Oct 31 03:19:18.075502 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:18.075422    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:18.075600 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:18.075565    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nb9d" podUID="8d251d9b-c5e7-46e7-b5ab-524b8171d83b"
Oct 31 03:19:18.075903 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:18.075864    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rtp6w"]
Oct 31 03:19:18.076071 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:18.076053    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:19:18.076177 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:18.076157    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtp6w" podUID="bf1633f9-bdb5-4a26-bd80-a83a4b5587e8"
Oct 31 03:19:18.076448 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:18.076428    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bx98q"]
Oct 31 03:19:18.076528 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:18.076520    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:19:18.076613 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:18.076597    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx98q" podUID="45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9"
Oct 31 03:19:20.113295 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:20.113268    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:19:20.113607 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:20.113278    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:19:20.113607 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:20.113393    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:20.113607 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:20.113394    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx98q" podUID="45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9"
Oct 31 03:19:20.113607 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:20.113494    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtp6w" podUID="bf1633f9-bdb5-4a26-bd80-a83a4b5587e8"
Oct 31 03:19:20.113607 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:20.113571    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nb9d" podUID="8d251d9b-c5e7-46e7-b5ab-524b8171d83b"
Oct 31 03:19:22.113801 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:22.113713    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:19:22.113801 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:22.113736    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:19:22.114314 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:22.113821    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rtp6w" podUID="bf1633f9-bdb5-4a26-bd80-a83a4b5587e8"
Oct 31 03:19:22.114314 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:22.113880    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx98q" podUID="45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9"
Oct 31 03:19:22.114314 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:22.113930    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:22.114314 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:22.114013    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nb9d" podUID="8d251d9b-c5e7-46e7-b5ab-524b8171d83b"
Oct 31 03:19:23.945451 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:23.945426    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeReady"
Oct 31 03:19:23.945828 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:23.945553    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 03:19:24.060698 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.060613    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zxbcc"]
Oct 31 03:19:24.070232 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.070208    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qbl5h"]
Oct 31 03:19:24.070369 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.070347    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:24.073388 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.073368    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r9prp"]
Oct 31 03:19:24.073506 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.073489    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:24.076168 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.076147    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r9prp"
Oct 31 03:19:24.113752 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.113717    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:19:24.113752 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.113742    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:24.113752 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.113758    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:19:24.122675 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.122652    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 03:19:24.122818 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.122726    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 03:19:24.122818 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.122756    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-btzr7\""
Oct 31 03:19:24.130144 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.130118    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dbgqm\""
Oct 31 03:19:24.147668 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.147640    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r9prp"]
Oct 31 03:19:24.150674 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.150648    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zxbcc"]
Oct 31 03:19:24.151263 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.151243    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qbl5h"]
Oct 31 03:19:24.160868 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.160845    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac9dbe77-127f-4a88-9634-82560e6424ce-tmp-dir\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:24.160997 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.160875    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/be4c5472-30cf-447a-85fe-b90f5f8b022a-data-volume\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:24.160997 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.160912    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7360ff6-d326-485b-bd3d-fd2f0f9099b5-cert\") pod \"ingress-canary-r9prp\" (UID: \"a7360ff6-d326-485b-bd3d-fd2f0f9099b5\") " pod="openshift-ingress-canary/ingress-canary-r9prp"
Oct 31 03:19:24.161061 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.161020    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:24.161061 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.161042    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be4c5472-30cf-447a-85fe-b90f5f8b022a-crio-socket\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:24.161120 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.161060    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:24.161150 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.161120    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz5wr\" (UniqueName: \"kubernetes.io/projected/a7360ff6-d326-485b-bd3d-fd2f0f9099b5-kube-api-access-kz5wr\") pod \"ingress-canary-r9prp\" (UID: \"a7360ff6-d326-485b-bd3d-fd2f0f9099b5\") " pod="openshift-ingress-canary/ingress-canary-r9prp"
Oct 31 03:19:24.161204 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.161191    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxphc\" (UniqueName: \"kubernetes.io/projected/ac9dbe77-127f-4a88-9634-82560e6424ce-kube-api-access-kxphc\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:24.161237 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.161213    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ac9dbe77-127f-4a88-9634-82560e6424ce-config-volume\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:24.161237 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.161227    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be4c5472-30cf-447a-85fe-b90f5f8b022a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:24.161310 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.161294    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ln55\" (UniqueName: \"kubernetes.io/projected/be4c5472-30cf-447a-85fe-b90f5f8b022a-kube-api-access-5ln55\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:24.223315 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.223270    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 03:19:24.223315 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.223307    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 03:19:24.223315 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.223270    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 03:19:24.223596 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.223343    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 03:19:24.223596 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.223379    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 03:19:24.223596 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.223426    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 03:19:24.223735 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.223621    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 03:19:24.223735 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.223696    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-vxflf\""
Oct 31 03:19:24.223735 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.223708    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 03:19:24.223735 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.223715    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 03:19:24.223735 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.223717    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-n6jvn\""
Oct 31 03:19:24.223950 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.223744    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 03:19:24.223950 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.223754    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hkxjs\""
Oct 31 03:19:24.223950 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.223711    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 03:19:24.223950 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.223816    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v5ddh\""
Oct 31 03:19:24.262057 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.262026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ln55\" (UniqueName: \"kubernetes.io/projected/be4c5472-30cf-447a-85fe-b90f5f8b022a-kube-api-access-5ln55\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:24.262145 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.262079    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac9dbe77-127f-4a88-9634-82560e6424ce-tmp-dir\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:24.262145 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.262098    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/be4c5472-30cf-447a-85fe-b90f5f8b022a-data-volume\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:24.262246 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.262217    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7360ff6-d326-485b-bd3d-fd2f0f9099b5-cert\") pod \"ingress-canary-r9prp\" (UID: \"a7360ff6-d326-485b-bd3d-fd2f0f9099b5\") " pod="openshift-ingress-canary/ingress-canary-r9prp"
Oct 31 03:19:24.262289 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.262257    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:24.262289 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.262278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be4c5472-30cf-447a-85fe-b90f5f8b022a-crio-socket\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:24.262376 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.262301    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:24.262376 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.262347    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz5wr\" (UniqueName: \"kubernetes.io/projected/a7360ff6-d326-485b-bd3d-fd2f0f9099b5-kube-api-access-kz5wr\") pod \"ingress-canary-r9prp\" (UID: \"a7360ff6-d326-485b-bd3d-fd2f0f9099b5\") " pod="openshift-ingress-canary/ingress-canary-r9prp"
Oct 31 03:19:24.262467 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.262373    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxphc\" (UniqueName: \"kubernetes.io/projected/ac9dbe77-127f-4a88-9634-82560e6424ce-kube-api-access-kxphc\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:24.262467 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.262410    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ac9dbe77-127f-4a88-9634-82560e6424ce-config-volume\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:24.262467 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:24.262411    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:19:24.262467 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.262454    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/be4c5472-30cf-447a-85fe-b90f5f8b022a-data-volume\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:24.262647 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:24.262480    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls podName:ac9dbe77-127f-4a88-9634-82560e6424ce nodeName:}" failed. No retries permitted until 2026-10-31 03:19:24.762459749 +0000 UTC m=+30.130947080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls") pod "dns-default-zxbcc" (UID: "ac9dbe77-127f-4a88-9634-82560e6424ce") : secret "dns-default-metrics-tls" not found
Oct 31 03:19:24.262647 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:24.262495    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:19:24.262647 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.262454    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac9dbe77-127f-4a88-9634-82560e6424ce-tmp-dir\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:24.262647 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:24.262547    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls podName:be4c5472-30cf-447a-85fe-b90f5f8b022a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:24.762530259 +0000 UTC m=+30.131017599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qbl5h" (UID: "be4c5472-30cf-447a-85fe-b90f5f8b022a") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:19:24.262647 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.262498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be4c5472-30cf-447a-85fe-b90f5f8b022a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:24.262647 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:24.262555    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 03:19:24.262647 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:24.262630    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be4c5472-30cf-447a-85fe-b90f5f8b022a-kube-rbac-proxy-cm podName:be4c5472-30cf-447a-85fe-b90f5f8b022a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:24.762619607 +0000 UTC m=+30.131106955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/be4c5472-30cf-447a-85fe-b90f5f8b022a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qbl5h" (UID: "be4c5472-30cf-447a-85fe-b90f5f8b022a") : configmap "kube-rbac-proxy" not found
Oct 31 03:19:24.262913 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.262684    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be4c5472-30cf-447a-85fe-b90f5f8b022a-crio-socket\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:24.263034 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.263018    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ac9dbe77-127f-4a88-9634-82560e6424ce-config-volume\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:24.266367 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.266345    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7360ff6-d326-485b-bd3d-fd2f0f9099b5-cert\") pod \"ingress-canary-r9prp\" (UID: \"a7360ff6-d326-485b-bd3d-fd2f0f9099b5\") " pod="openshift-ingress-canary/ingress-canary-r9prp"
Oct 31 03:19:24.324350 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.324276    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxphc\" (UniqueName: \"kubernetes.io/projected/ac9dbe77-127f-4a88-9634-82560e6424ce-kube-api-access-kxphc\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:24.324483 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.324389    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz5wr\" (UniqueName: \"kubernetes.io/projected/a7360ff6-d326-485b-bd3d-fd2f0f9099b5-kube-api-access-kz5wr\") pod \"ingress-canary-r9prp\" (UID: \"a7360ff6-d326-485b-bd3d-fd2f0f9099b5\") " pod="openshift-ingress-canary/ingress-canary-r9prp"
Oct 31 03:19:24.324483 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.324400    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ln55\" (UniqueName: \"kubernetes.io/projected/be4c5472-30cf-447a-85fe-b90f5f8b022a-kube-api-access-5ln55\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:24.385777 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.385747    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-tqfr6"]
Oct 31 03:19:24.389766 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.389749    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-tqfr6"
Oct 31 03:19:24.394550 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.394518    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r9prp"
Oct 31 03:19:24.445016 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.444960    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 03:19:24.445016 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.444960    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 03:19:24.445016 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.445002    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vknbj\""
Oct 31 03:19:24.445317 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.445063    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 03:19:24.447241 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.446074    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 03:19:24.448949 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.447559    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-tqfr6"]
Oct 31 03:19:24.463700 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.463671    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a0f3be01-6bec-4a39-980a-e6342fcb3624-signing-cabundle\") pod \"service-ca-5fffc97bdf-tqfr6\" (UID: \"a0f3be01-6bec-4a39-980a-e6342fcb3624\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tqfr6"
Oct 31 03:19:24.463903 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.463758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a0f3be01-6bec-4a39-980a-e6342fcb3624-signing-key\") pod \"service-ca-5fffc97bdf-tqfr6\" (UID: \"a0f3be01-6bec-4a39-980a-e6342fcb3624\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tqfr6"
Oct 31 03:19:24.463903 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.463821    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjh4p\" (UniqueName: \"kubernetes.io/projected/a0f3be01-6bec-4a39-980a-e6342fcb3624-kube-api-access-cjh4p\") pod \"service-ca-5fffc97bdf-tqfr6\" (UID: \"a0f3be01-6bec-4a39-980a-e6342fcb3624\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tqfr6"
Oct 31 03:19:24.564392 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.564360    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a0f3be01-6bec-4a39-980a-e6342fcb3624-signing-key\") pod \"service-ca-5fffc97bdf-tqfr6\" (UID: \"a0f3be01-6bec-4a39-980a-e6342fcb3624\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tqfr6"
Oct 31 03:19:24.564561 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.564420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjh4p\" (UniqueName: \"kubernetes.io/projected/a0f3be01-6bec-4a39-980a-e6342fcb3624-kube-api-access-cjh4p\") pod \"service-ca-5fffc97bdf-tqfr6\" (UID: \"a0f3be01-6bec-4a39-980a-e6342fcb3624\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tqfr6"
Oct 31 03:19:24.564633 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.564604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a0f3be01-6bec-4a39-980a-e6342fcb3624-signing-cabundle\") pod \"service-ca-5fffc97bdf-tqfr6\" (UID: \"a0f3be01-6bec-4a39-980a-e6342fcb3624\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tqfr6"
Oct 31 03:19:24.565243 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.565226    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a0f3be01-6bec-4a39-980a-e6342fcb3624-signing-cabundle\") pod \"service-ca-5fffc97bdf-tqfr6\" (UID: \"a0f3be01-6bec-4a39-980a-e6342fcb3624\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tqfr6"
Oct 31 03:19:24.566800 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.566784    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a0f3be01-6bec-4a39-980a-e6342fcb3624-signing-key\") pod \"service-ca-5fffc97bdf-tqfr6\" (UID: \"a0f3be01-6bec-4a39-980a-e6342fcb3624\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tqfr6"
Oct 31 03:19:24.625651 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.625588    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjh4p\" (UniqueName: \"kubernetes.io/projected/a0f3be01-6bec-4a39-980a-e6342fcb3624-kube-api-access-cjh4p\") pod \"service-ca-5fffc97bdf-tqfr6\" (UID: \"a0f3be01-6bec-4a39-980a-e6342fcb3624\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tqfr6"
Oct 31 03:19:24.700300 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.700268    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-tqfr6"
Oct 31 03:19:24.766147 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.766120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:24.766273 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.766152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:24.766273 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.766182    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be4c5472-30cf-447a-85fe-b90f5f8b022a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:24.766273 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:24.766250    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 03:19:24.766368 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:24.766269    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:19:24.766368 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:24.766271    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:19:24.766368 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:24.766313    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be4c5472-30cf-447a-85fe-b90f5f8b022a-kube-rbac-proxy-cm podName:be4c5472-30cf-447a-85fe-b90f5f8b022a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:25.766297108 +0000 UTC m=+31.134784430 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/be4c5472-30cf-447a-85fe-b90f5f8b022a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qbl5h" (UID: "be4c5472-30cf-447a-85fe-b90f5f8b022a") : configmap "kube-rbac-proxy" not found
Oct 31 03:19:24.766368 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:24.766325    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls podName:be4c5472-30cf-447a-85fe-b90f5f8b022a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:25.766319772 +0000 UTC m=+31.134807091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qbl5h" (UID: "be4c5472-30cf-447a-85fe-b90f5f8b022a") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:19:24.766368 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:24.766334    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls podName:ac9dbe77-127f-4a88-9634-82560e6424ce nodeName:}" failed. No retries permitted until 2026-10-31 03:19:25.766329434 +0000 UTC m=+31.134816752 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls") pod "dns-default-zxbcc" (UID: "ac9dbe77-127f-4a88-9634-82560e6424ce") : secret "dns-default-metrics-tls" not found
Oct 31 03:19:24.771272 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:24.771250    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r9prp"]
Oct 31 03:19:24.774352 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:19:24.774326    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7360ff6_d326_485b_bd3d_fd2f0f9099b5.slice/crio-f80f3ff5d1cb2d8af42e4ca1df00bb5cd38c4aca3ac41c988f141e40f8c30ff7 WatchSource:0}: Error finding container f80f3ff5d1cb2d8af42e4ca1df00bb5cd38c4aca3ac41c988f141e40f8c30ff7: Status 404 returned error can't find the container with id f80f3ff5d1cb2d8af42e4ca1df00bb5cd38c4aca3ac41c988f141e40f8c30ff7
Oct 31 03:19:25.065512 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:25.065440    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-tqfr6"]
Oct 31 03:19:25.068267 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:19:25.068242    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f3be01_6bec_4a39_980a_e6342fcb3624.slice/crio-f42f3f365dbd3ecb6399cde9c9c656fe561a2142f8431d37660ad50902ff8a41 WatchSource:0}: Error finding container f42f3f365dbd3ecb6399cde9c9c656fe561a2142f8431d37660ad50902ff8a41: Status 404 returned error can't find the container with id f42f3f365dbd3ecb6399cde9c9c656fe561a2142f8431d37660ad50902ff8a41
Oct 31 03:19:25.278307 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:25.278275    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-tqfr6" event={"ID":"a0f3be01-6bec-4a39-980a-e6342fcb3624","Type":"ContainerStarted","Data":"f42f3f365dbd3ecb6399cde9c9c656fe561a2142f8431d37660ad50902ff8a41"}
Oct 31 03:19:25.279432 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:25.279327    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r9prp" event={"ID":"a7360ff6-d326-485b-bd3d-fd2f0f9099b5","Type":"ContainerStarted","Data":"f80f3ff5d1cb2d8af42e4ca1df00bb5cd38c4aca3ac41c988f141e40f8c30ff7"}
Oct 31 03:19:25.613395 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:25.613361    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kqgxt"
Oct 31 03:19:25.613573 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:25.613541    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 03:19:25.613978 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:25.613951    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kqgxt"
Oct 31 03:19:25.773784 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:25.773749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:25.773784 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:25.773784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:25.774048 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:25.773817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be4c5472-30cf-447a-85fe-b90f5f8b022a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:25.774048 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:25.773935    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:19:25.774048 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:25.773962    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 03:19:25.774048 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:25.773936    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:19:25.774048 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:25.774015    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls podName:be4c5472-30cf-447a-85fe-b90f5f8b022a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:27.773995675 +0000 UTC m=+33.142482993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qbl5h" (UID: "be4c5472-30cf-447a-85fe-b90f5f8b022a") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:19:25.774048 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:25.774037    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls podName:ac9dbe77-127f-4a88-9634-82560e6424ce nodeName:}" failed. No retries permitted until 2026-10-31 03:19:27.774023931 +0000 UTC m=+33.142511250 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls") pod "dns-default-zxbcc" (UID: "ac9dbe77-127f-4a88-9634-82560e6424ce") : secret "dns-default-metrics-tls" not found
Oct 31 03:19:25.774048 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:25.774055    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be4c5472-30cf-447a-85fe-b90f5f8b022a-kube-rbac-proxy-cm podName:be4c5472-30cf-447a-85fe-b90f5f8b022a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:27.774045728 +0000 UTC m=+33.142533048 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/be4c5472-30cf-447a-85fe-b90f5f8b022a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qbl5h" (UID: "be4c5472-30cf-447a-85fe-b90f5f8b022a") : configmap "kube-rbac-proxy" not found
Oct 31 03:19:27.791700 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:27.791673    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:27.792107 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:27.791705    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:27.792107 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:27.791739    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be4c5472-30cf-447a-85fe-b90f5f8b022a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:27.792107 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:27.791837    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 03:19:27.792107 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:27.791915    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be4c5472-30cf-447a-85fe-b90f5f8b022a-kube-rbac-proxy-cm podName:be4c5472-30cf-447a-85fe-b90f5f8b022a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:31.791898836 +0000 UTC m=+37.160386167 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/be4c5472-30cf-447a-85fe-b90f5f8b022a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qbl5h" (UID: "be4c5472-30cf-447a-85fe-b90f5f8b022a") : configmap "kube-rbac-proxy" not found
Oct 31 03:19:27.792107 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:27.791997    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:19:27.792107 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:27.792041    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls podName:ac9dbe77-127f-4a88-9634-82560e6424ce nodeName:}" failed. No retries permitted until 2026-10-31 03:19:31.792027735 +0000 UTC m=+37.160515069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls") pod "dns-default-zxbcc" (UID: "ac9dbe77-127f-4a88-9634-82560e6424ce") : secret "dns-default-metrics-tls" not found
Oct 31 03:19:27.792107 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:27.792040    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:19:27.792335 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:27.792115    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls podName:be4c5472-30cf-447a-85fe-b90f5f8b022a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:31.792095646 +0000 UTC m=+37.160582967 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qbl5h" (UID: "be4c5472-30cf-447a-85fe-b90f5f8b022a") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:19:28.094440 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:28.094358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs\") pod \"network-metrics-daemon-bx98q\" (UID: \"45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9\") " pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:19:28.094440 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:28.094403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-original-pull-secret\") pod \"global-pull-secret-syncer-7nb9d\" (UID: \"8d251d9b-c5e7-46e7-b5ab-524b8171d83b\") " pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:28.094612 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:28.094518    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 03:19:28.094612 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:28.094583    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs podName:45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9 nodeName:}" failed. No retries permitted until 2026-10-31 03:20:00.094568855 +0000 UTC m=+65.463056173 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs") pod "network-metrics-daemon-bx98q" (UID: "45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9") : secret "metrics-daemon-secret" not found
Oct 31 03:19:28.096815 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:28.096798    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-original-pull-secret\") pod \"global-pull-secret-syncer-7nb9d\" (UID: \"8d251d9b-c5e7-46e7-b5ab-524b8171d83b\") " pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:28.195056 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:28.195017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5jxx\" (UniqueName: \"kubernetes.io/projected/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8-kube-api-access-p5jxx\") pod \"network-check-target-rtp6w\" (UID: \"bf1633f9-bdb5-4a26-bd80-a83a4b5587e8\") " pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:19:28.197552 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:28.197526    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5jxx\" (UniqueName: \"kubernetes.io/projected/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8-kube-api-access-p5jxx\") pod \"network-check-target-rtp6w\" (UID: \"bf1633f9-bdb5-4a26-bd80-a83a4b5587e8\") " pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:19:28.288454 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:28.288422    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-tqfr6" event={"ID":"a0f3be01-6bec-4a39-980a-e6342fcb3624","Type":"ContainerStarted","Data":"da7bedf23fa378d97e918a27570a3d0e3f4e37a74d34c0568ee69ec2f40658ed"}
Oct 31 03:19:28.289983 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:28.289952    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r9prp" event={"ID":"a7360ff6-d326-485b-bd3d-fd2f0f9099b5","Type":"ContainerStarted","Data":"1f6203183fcca0aa9638a966a44fc81741235239aa65567d441d25796227df10"}
Oct 31 03:19:28.330825 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:28.330797    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:19:28.335778 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:28.335763    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:28.409125 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:28.409080    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-tqfr6" podStartSLOduration=1.79494025 podStartE2EDuration="4.409064901s" totalImagesPullingTime="2.614124651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:28.407827695 +0000 UTC m=+33.776315035" watchObservedRunningTime="2026-10-31 03:19:28.409064901 +0000 UTC m=+33.777552271"
Oct 31 03:19:28.524533 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:28.524476    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r9prp" podStartSLOduration=1.441932457 podStartE2EDuration="4.524458885s" totalImagesPullingTime="3.082526428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:28.523599461 +0000 UTC m=+33.892086800" watchObservedRunningTime="2026-10-31 03:19:28.524458885 +0000 UTC m=+33.892946225"
Oct 31 03:19:28.705658 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:28.705586    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7nb9d"]
Oct 31 03:19:28.708899 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:19:28.708861    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d251d9b_c5e7_46e7_b5ab_524b8171d83b.slice/crio-d61cdec76693cad5f1f6a9093e9d96d92957e4e6bb5f94af9ed9c4c5e51024a2 WatchSource:0}: Error finding container d61cdec76693cad5f1f6a9093e9d96d92957e4e6bb5f94af9ed9c4c5e51024a2: Status 404 returned error can't find the container with id d61cdec76693cad5f1f6a9093e9d96d92957e4e6bb5f94af9ed9c4c5e51024a2
Oct 31 03:19:28.755396 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:28.755362    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rtp6w"]
Oct 31 03:19:28.758460 ip-10-0-134-255 kubenswrapper[2577]: W1031 03:19:28.758432    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1633f9_bdb5_4a26_bd80_a83a4b5587e8.slice/crio-17e3b9117877b2bdb85084852dbe0e02a35b7a475587990611ffd6d46c70f4c7 WatchSource:0}: Error finding container 17e3b9117877b2bdb85084852dbe0e02a35b7a475587990611ffd6d46c70f4c7: Status 404 returned error can't find the container with id 17e3b9117877b2bdb85084852dbe0e02a35b7a475587990611ffd6d46c70f4c7
Oct 31 03:19:29.294079 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:29.294045    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7nb9d" event={"ID":"8d251d9b-c5e7-46e7-b5ab-524b8171d83b","Type":"ContainerStarted","Data":"d61cdec76693cad5f1f6a9093e9d96d92957e4e6bb5f94af9ed9c4c5e51024a2"}
Oct 31 03:19:29.295187 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:29.295162    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rtp6w" event={"ID":"bf1633f9-bdb5-4a26-bd80-a83a4b5587e8","Type":"ContainerStarted","Data":"17e3b9117877b2bdb85084852dbe0e02a35b7a475587990611ffd6d46c70f4c7"}
Oct 31 03:19:29.855939 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:29.855909    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j86qh_65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce/dns-node-resolver/0.log"
Oct 31 03:19:30.856749 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:30.856710    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2kzdk_8da6593d-d284-438d-96ee-ec0d9ddf1124/node-ca/0.log"
Oct 31 03:19:31.827410 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:31.827373    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be4c5472-30cf-447a-85fe-b90f5f8b022a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:31.827563 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:31.827466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:31.827563 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:31.827494    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:31.827695 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:31.827613    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:19:31.827695 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:31.827618    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:19:31.827695 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:31.827682    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls podName:ac9dbe77-127f-4a88-9634-82560e6424ce nodeName:}" failed. No retries permitted until 2026-10-31 03:19:39.827666942 +0000 UTC m=+45.196154263 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls") pod "dns-default-zxbcc" (UID: "ac9dbe77-127f-4a88-9634-82560e6424ce") : secret "dns-default-metrics-tls" not found
Oct 31 03:19:31.827843 ip-10-0-134-255 kubenswrapper[2577]: E1031 03:19:31.827702    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls podName:be4c5472-30cf-447a-85fe-b90f5f8b022a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:39.827694708 +0000 UTC m=+45.196182027 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qbl5h" (UID: "be4c5472-30cf-447a-85fe-b90f5f8b022a") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:19:31.828087 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:31.828064    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be4c5472-30cf-447a-85fe-b90f5f8b022a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:31.855614 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:31.855582    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r9prp_a7360ff6-d326-485b-bd3d-fd2f0f9099b5/serve-healthcheck-canary/0.log"
Oct 31 03:19:32.195830 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:32.195721    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kqgxt_ff0da67d-c75e-4487-ba6a-b85e04a496d9/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:32.585622 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:32.585530    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kqgxt_ff0da67d-c75e-4487-ba6a-b85e04a496d9/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:32.792247 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:32.792221    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kqgxt_ff0da67d-c75e-4487-ba6a-b85e04a496d9/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:33.048953 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:33.048859    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kqgxt_ff0da67d-c75e-4487-ba6a-b85e04a496d9/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:33.256150 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:33.256121    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kqgxt_ff0da67d-c75e-4487-ba6a-b85e04a496d9/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:33.307704 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:33.307625    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rtp6w" event={"ID":"bf1633f9-bdb5-4a26-bd80-a83a4b5587e8","Type":"ContainerStarted","Data":"025aebfe2e2677cc5e0734276cc246cb8f2c09ee1e7733473c707cfa713f6ccf"}
Oct 31 03:19:33.307875 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:33.307705    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:19:33.423825 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:33.423756    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rtp6w" podStartSLOduration=34.453542413 podStartE2EDuration="38.423741935s" totalImagesPullingTime="3.970199522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:33.422492645 +0000 UTC m=+38.790979985" watchObservedRunningTime="2026-10-31 03:19:33.423741935 +0000 UTC m=+38.792229276"
Oct 31 03:19:34.311987 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:34.311947    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7nb9d" event={"ID":"8d251d9b-c5e7-46e7-b5ab-524b8171d83b","Type":"ContainerStarted","Data":"2d2266618f2d49a089e3aec82f4934e6521b8067b98b13b4d21b4420ef887186"}
Oct 31 03:19:34.402121 ip-10-0-134-255 kubenswrapper[2577]: I1031 03:19:34.402017    2577 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 03:19:34.402184 ip-10-0-134-255 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 03:19:34.408548 ip-10-0-134-255 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 03:19:34.408773 ip-10-0-134-255 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 03:19:34.408837 ip-10-0-134-255 systemd[1]: kubelet.service: Consumed 2.407s CPU time.
Oct 31 03:19:34.419630 ip-10-0-134-255 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 03:19:34.518927 ip-10-0-134-255 kubenswrapper[4447]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:19:34.518927 ip-10-0-134-255 kubenswrapper[4447]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 03:19:34.518927 ip-10-0-134-255 kubenswrapper[4447]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:19:34.518927 ip-10-0-134-255 kubenswrapper[4447]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:19:34.521040 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.520951    4447 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:19:34.521040 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521040    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:19:34.521040 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521044    4447 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:19:34.521136 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521048    4447 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:19:34.521136 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521050    4447 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:19:34.521136 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521053    4447 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:19:34.521136 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521056    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:19:34.521136 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521058    4447 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:19:34.521136 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521069    4447 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:19:34.521136 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521072    4447 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:19:34.521136 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521074    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:19:34.521136 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521077    4447 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:19:34.521136 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521079    4447 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:19:34.521136 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521082    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:19:34.521136 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521084    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:19:34.521136 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521086    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:19:34.521136 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521089    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:19:34.521136 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521091    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:19:34.521136 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521094    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:19:34.521136 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521097    4447 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:19:34.521136 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521099    4447 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:19:34.521136 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521102    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:19:34.521136 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521104    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:19:34.521636 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521109    4447 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:19:34.521636 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521113    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:19:34.521636 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521116    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:19:34.521636 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521119    4447 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:19:34.521636 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521121    4447 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:19:34.521636 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521124    4447 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:19:34.521636 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521126    4447 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:19:34.521636 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521129    4447 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:19:34.521636 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521132    4447 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:19:34.521636 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521134    4447 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:19:34.521636 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521137    4447 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:19:34.521636 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521140    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:19:34.521636 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521142    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:19:34.521636 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521145    4447 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:19:34.521636 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521148    4447 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:19:34.521636 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521151    4447 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:19:34.521636 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521154    4447 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:19:34.521636 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521156    4447 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:19:34.521636 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521159    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:19:34.522324 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521161    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:19:34.522324 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521164    4447 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:19:34.522324 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521168    4447 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:19:34.522324 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521171    4447 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:19:34.522324 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521175    4447 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:19:34.522324 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521178    4447 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:19:34.522324 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521180    4447 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:19:34.522324 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521183    4447 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:19:34.522324 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521185    4447 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:19:34.522324 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521188    4447 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:19:34.522324 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521191    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:19:34.522324 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521193    4447 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:19:34.522324 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521196    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:19:34.522324 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521198    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:19:34.522324 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521201    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:19:34.522324 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521204    4447 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:19:34.522324 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521206    4447 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:19:34.522324 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521209    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:19:34.522324 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521211    4447 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:19:34.522798 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521213    4447 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:19:34.522798 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521216    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:19:34.522798 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521218    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:19:34.522798 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521221    4447 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:19:34.522798 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521223    4447 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:19:34.522798 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521226    4447 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:19:34.522798 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521228    4447 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:19:34.522798 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521230    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:19:34.522798 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521233    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:19:34.522798 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521235    4447 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:19:34.522798 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521243    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:19:34.522798 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521246    4447 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:19:34.522798 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521248    4447 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:19:34.522798 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521251    4447 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:19:34.522798 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521253    4447 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:19:34.522798 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521255    4447 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:19:34.522798 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521258    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:19:34.522798 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521260    4447 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:19:34.522798 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521263    4447 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:19:34.522798 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521265    4447 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:19:34.523303 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521267    4447 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:19:34.523303 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521270    4447 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:19:34.523303 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521272    4447 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:19:34.523303 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521275    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:19:34.523303 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521278    4447 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:19:34.523303 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521280    4447 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:19:34.523303 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521282    4447 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:19:34.523303 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521286    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:19:34.523303 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521288    4447 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:19:34.523303 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521291    4447 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:19:34.523303 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521293    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:19:34.523303 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521296    4447 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:19:34.523303 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521298    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:19:34.523303 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521300    4447 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:19:34.523303 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521303    4447 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:19:34.523303 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521305    4447 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:19:34.523303 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521307    4447 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:19:34.523303 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521310    4447 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:19:34.523303 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521312    4447 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:19:34.523774 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521314    4447 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:19:34.523774 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521317    4447 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:19:34.523774 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521319    4447 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:19:34.523774 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521321    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:19:34.523774 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521324    4447 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:19:34.523774 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.521327    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:19:34.523774 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522414    4447 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:19:34.523774 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522423    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:19:34.523774 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522426    4447 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:19:34.523774 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522429    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:19:34.523774 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522432    4447 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:19:34.523774 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522436    4447 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:19:34.523774 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522438    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:19:34.523774 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522440    4447 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:19:34.523774 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522443    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:19:34.523774 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522446    4447 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:19:34.523774 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522449    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:19:34.523774 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522451    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:19:34.523774 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522454    4447 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:19:34.523774 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522456    4447 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:19:34.524289 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522459    4447 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:19:34.524289 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522462    4447 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:19:34.524289 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522464    4447 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:19:34.524289 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522467    4447 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:19:34.524289 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522470    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:19:34.524289 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522473    4447 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:19:34.524289 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522477    4447 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:19:34.524289 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522481    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:19:34.524289 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522485    4447 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:19:34.524289 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522487    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:19:34.524289 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522490    4447 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:19:34.524289 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522493    4447 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:19:34.524289 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522496    4447 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:19:34.524289 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522498    4447 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:19:34.524289 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522501    4447 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:19:34.524289 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522503    4447 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:19:34.524289 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522506    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:19:34.524289 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522509    4447 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:19:34.524289 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522513    4447 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:19:34.524289 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522516    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:19:34.524775 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522518    4447 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:19:34.524775 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522521    4447 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:19:34.524775 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522524    4447 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:19:34.524775 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522526    4447 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:19:34.524775 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522528    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:19:34.524775 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522531    4447 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:19:34.524775 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522533    4447 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:19:34.524775 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522536    4447 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:19:34.524775 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522539    4447 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:19:34.524775 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522541    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:19:34.524775 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522544    4447 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:19:34.524775 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522546    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:19:34.524775 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522549    4447 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:19:34.524775 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522551    4447 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:19:34.524775 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522554    4447 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:19:34.524775 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522556    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:19:34.524775 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522565    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:19:34.524775 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522567    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:19:34.524775 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522570    4447 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:19:34.524775 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522572    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:19:34.525472 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522575    4447 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:19:34.525472 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522577    4447 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:19:34.525472 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522580    4447 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:19:34.525472 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522582    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:19:34.525472 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522585    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:19:34.525472 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522587    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:19:34.525472 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522589    4447 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:19:34.525472 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522592    4447 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:19:34.525472 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522594    4447 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:19:34.525472 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522597    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:19:34.525472 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522600    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:19:34.525472 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522603    4447 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:19:34.525472 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522605    4447 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:19:34.525472 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522608    4447 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:19:34.525472 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522610    4447 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:19:34.525472 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522613    4447 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:19:34.525472 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522615    4447 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:19:34.525472 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522618    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:19:34.525472 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522620    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:19:34.525953 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522623    4447 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:19:34.525953 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522625    4447 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:19:34.525953 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522627    4447 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:19:34.525953 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522630    4447 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:19:34.525953 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522632    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:19:34.525953 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522635    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:19:34.525953 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522637    4447 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:19:34.525953 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522640    4447 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:19:34.525953 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522642    4447 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:19:34.525953 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522644    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:19:34.525953 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522647    4447 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:19:34.525953 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522649    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:19:34.525953 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522652    4447 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:19:34.525953 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522655    4447 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:19:34.525953 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522657    4447 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:19:34.525953 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522659    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:19:34.525953 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522662    4447 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:19:34.525953 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522665    4447 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:19:34.525953 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522669    4447 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:19:34.526432 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522672    4447 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:19:34.526432 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522675    4447 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:19:34.526432 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522677    4447 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:19:34.526432 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522679    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:19:34.526432 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522682    4447 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:19:34.526432 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522685    4447 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:19:34.526432 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522688    4447 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:19:34.526432 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522690    4447 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:19:34.526432 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522693    4447 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:19:34.526432 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522695    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:19:34.526432 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522697    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:19:34.526432 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522700    4447 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:19:34.526432 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522703    4447 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:19:34.526432 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.522705    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:19:34.526432 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523208    4447 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:19:34.526432 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523216    4447 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:19:34.526432 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523222    4447 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:19:34.526432 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523226    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:19:34.526432 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523228    4447 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:19:34.526432 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523231    4447 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:19:34.526932 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523234    4447 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:19:34.526932 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523236    4447 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:19:34.526932 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523239    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:19:34.526932 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523241    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:19:34.526932 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523244    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:19:34.526932 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523247    4447 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:19:34.526932 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523249    4447 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:19:34.526932 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523252    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:19:34.526932 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523254    4447 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:19:34.526932 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523257    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:19:34.526932 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523259    4447 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:19:34.526932 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523261    4447 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:19:34.526932 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523264    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:19:34.526932 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523267    4447 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:19:34.526932 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523269    4447 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:19:34.526932 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523272    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:19:34.526932 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523274    4447 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:19:34.526932 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523277    4447 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:19:34.526932 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523280    4447 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:19:34.526932 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523282    4447 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:19:34.527639 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523285    4447 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:19:34.527639 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523288    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:19:34.527639 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523290    4447 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:19:34.527639 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523292    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:19:34.527639 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523295    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:19:34.527639 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523297    4447 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:19:34.527639 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523299    4447 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:19:34.527639 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523302    4447 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:19:34.527639 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523306    4447 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:19:34.527639 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523308    4447 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:19:34.527639 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523311    4447 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:19:34.527639 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523314    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:19:34.527639 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523318    4447 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:19:34.527639 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523322    4447 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:19:34.527639 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523325    4447 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:19:34.527639 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523327    4447 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:19:34.527639 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523330    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:19:34.527639 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523332    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:19:34.527639 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523334    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:19:34.528390 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523337    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:19:34.528390 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523339    4447 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:19:34.528390 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523342    4447 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:19:34.528390 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523344    4447 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:19:34.528390 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523346    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:19:34.528390 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523349    4447 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:19:34.528390 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523352    4447 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:19:34.528390 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523354    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:19:34.528390 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523356    4447 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:19:34.528390 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523359    4447 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:19:34.528390 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523361    4447 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:19:34.528390 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523364    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:19:34.528390 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523366    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:19:34.528390 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523369    4447 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:19:34.528390 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523371    4447 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:19:34.528390 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523373    4447 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:19:34.528390 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523375    4447 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:19:34.528390 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523378    4447 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:19:34.528390 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523380    4447 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:19:34.528390 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523383    4447 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:19:34.529159 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523385    4447 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:19:34.529159 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523388    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:19:34.529159 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523390    4447 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:19:34.529159 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523393    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:19:34.529159 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523396    4447 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:19:34.529159 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523398    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:19:34.529159 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523400    4447 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:19:34.529159 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523403    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:19:34.529159 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523405    4447 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:19:34.529159 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523408    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:19:34.529159 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523410    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:19:34.529159 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523412    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:19:34.529159 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523415    4447 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:19:34.529159 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523417    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:19:34.529159 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523419    4447 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:19:34.529159 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523421    4447 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:19:34.529159 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523424    4447 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:19:34.529159 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523426    4447 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:19:34.529159 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523428    4447 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:19:34.529618 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523431    4447 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:19:34.529618 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523433    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:19:34.529618 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523436    4447 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:19:34.529618 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523438    4447 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:19:34.529618 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523440    4447 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:19:34.529618 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523443    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:19:34.529618 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523446    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:19:34.529618 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523448    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:19:34.529618 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523450    4447 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:19:34.529618 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523452    4447 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:19:34.529618 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523455    4447 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:19:34.529618 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523457    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:19:34.529618 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523460    4447 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:19:34.529618 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523462    4447 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:19:34.529618 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523466    4447 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:19:34.529618 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523468    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:19:34.529618 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523470    4447 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:19:34.529618 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523473    4447 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:19:34.529618 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523475    4447 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:19:34.529618 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523477    4447 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:19:34.530120 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523480    4447 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:19:34.530120 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.523482    4447 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:19:34.530120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523582    4447 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 03:19:34.530120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523593    4447 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 03:19:34.530120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523606    4447 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 03:19:34.530120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523617    4447 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 03:19:34.530120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523622    4447 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 03:19:34.530120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523625    4447 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 03:19:34.530120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523630    4447 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 03:19:34.530120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523634    4447 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 03:19:34.530120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523637    4447 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 03:19:34.530120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523640    4447 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 03:19:34.530120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523643    4447 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 03:19:34.530120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523647    4447 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 03:19:34.530120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523650    4447 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 03:19:34.530120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523653    4447 flags.go:64] FLAG: --cgroup-root=""
Oct 31 03:19:34.530120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523656    4447 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 03:19:34.530120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523658    4447 flags.go:64] FLAG: --client-ca-file=""
Oct 31 03:19:34.530120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523662    4447 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 03:19:34.530120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523665    4447 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 03:19:34.530120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523668    4447 flags.go:64] FLAG: --cluster-domain=""
Oct 31 03:19:34.530120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523671    4447 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 03:19:34.530120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523674    4447 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 03:19:34.530120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523676    4447 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523680    4447 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523684    4447 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523687    4447 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523689    4447 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523693    4447 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523695    4447 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523699    4447 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523702    4447 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523706    4447 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523708    4447 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523713    4447 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523716    4447 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523719    4447 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523721    4447 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523724    4447 flags.go:64] FLAG: --enable-server="true"
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523727    4447 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523731    4447 flags.go:64] FLAG: --event-burst="100"
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523734    4447 flags.go:64] FLAG: --event-qps="50"
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523740    4447 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523743    4447 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523745    4447 flags.go:64] FLAG: --eviction-hard=""
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523750    4447 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523753    4447 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523756    4447 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 03:19:34.530698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523759    4447 flags.go:64] FLAG: --eviction-soft=""
Oct 31 03:19:34.531353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523762    4447 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 03:19:34.531353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523765    4447 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 03:19:34.531353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523768    4447 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 03:19:34.531353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523771    4447 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 03:19:34.531353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523774    4447 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 03:19:34.531353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523777    4447 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 03:19:34.531353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523779    4447 flags.go:64] FLAG: --feature-gates=""
Oct 31 03:19:34.531353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523784    4447 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 03:19:34.531353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523787    4447 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 03:19:34.531353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523790    4447 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 03:19:34.531353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523793    4447 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 03:19:34.531353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523796    4447 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 03:19:34.531353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523799    4447 flags.go:64] FLAG: --help="false"
Oct 31 03:19:34.531353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523803    4447 flags.go:64] FLAG: --hostname-override="ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:19:34.531353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523808    4447 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 03:19:34.531353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523811    4447 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 03:19:34.531353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523813    4447 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 03:19:34.531353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523817    4447 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 03:19:34.531353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523820    4447 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 03:19:34.531353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523823    4447 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 03:19:34.531353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523826    4447 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 03:19:34.531353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523829    4447 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 03:19:34.531353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523831    4447 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523834    4447 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523837    4447 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523842    4447 flags.go:64] FLAG: --kube-reserved=""
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523845    4447 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523847    4447 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523850    4447 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523853    4447 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523856    4447 flags.go:64] FLAG: --lock-file=""
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523858    4447 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523861    4447 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523864    4447 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523870    4447 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523873    4447 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523876    4447 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523894    4447 flags.go:64] FLAG: --logging-format="text"
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523900    4447 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523905    4447 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523910    4447 flags.go:64] FLAG: --manifest-url=""
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523913    4447 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523917    4447 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523920    4447 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523925    4447 flags.go:64] FLAG: --max-pods="110"
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523927    4447 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523931    4447 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 03:19:34.531913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523934    4447 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523937    4447 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523940    4447 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523943    4447 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523945    4447 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523952    4447 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523955    4447 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523958    4447 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523961    4447 flags.go:64] FLAG: --pod-cidr=""
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523964    4447 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523968    4447 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523971    4447 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523974    4447 flags.go:64] FLAG: --port="10250"
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523977    4447 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523980    4447 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0864006630e1e7726"
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523983    4447 flags.go:64] FLAG: --qos-reserved=""
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523986    4447 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523989    4447 flags.go:64] FLAG: --register-node="true"
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523992    4447 flags.go:64] FLAG: --register-with-taints=""
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523996    4447 flags.go:64] FLAG: --registry-burst="10"
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.523998    4447 flags.go:64] FLAG: --registry-qps="5"
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524002    4447 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524005    4447 flags.go:64] FLAG: --reserved-memory=""
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524009    4447 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524012    4447 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524015    4447 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 03:19:34.532637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524017    4447 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524020    4447 flags.go:64] FLAG: --runonce="false"
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524023    4447 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524025    4447 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524028    4447 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524032    4447 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524035    4447 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524038    4447 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524041    4447 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524044    4447 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524047    4447 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524050    4447 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524054    4447 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524057    4447 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524060    4447 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524062    4447 flags.go:64] FLAG: --system-cgroups=""
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524066    4447 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524071    4447 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524074    4447 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524077    4447 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524080    4447 flags.go:64] FLAG: --tls-min-version=""
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524091    4447 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524095    4447 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524098    4447 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524100    4447 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 03:19:34.533379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524103    4447 flags.go:64] FLAG: --v="2"
Oct 31 03:19:34.533999 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524108    4447 flags.go:64] FLAG: --version="false"
Oct 31 03:19:34.533999 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524112    4447 flags.go:64] FLAG: --vmodule=""
Oct 31 03:19:34.533999 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524116    4447 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 03:19:34.533999 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524120    4447 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 03:19:34.533999 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524253    4447 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:19:34.533999 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524256    4447 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:19:34.533999 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524259    4447 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:19:34.533999 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524262    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:19:34.533999 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524264    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:19:34.533999 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524267    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:19:34.533999 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524269    4447 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:19:34.533999 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524274    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:19:34.533999 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524276    4447 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:19:34.533999 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524279    4447 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:19:34.533999 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524281    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:19:34.533999 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524283    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:19:34.533999 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524286    4447 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:19:34.533999 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524288    4447 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:19:34.533999 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524291    4447 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:19:34.533999 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524293    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:19:34.534611 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524296    4447 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:19:34.534611 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524298    4447 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:19:34.534611 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524302    4447 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:19:34.534611 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524305    4447 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:19:34.534611 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524307    4447 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:19:34.534611 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524310    4447 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:19:34.534611 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524313    4447 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:19:34.534611 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524315    4447 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:19:34.534611 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524318    4447 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:19:34.534611 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524320    4447 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:19:34.534611 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524323    4447 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:19:34.534611 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524325    4447 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:19:34.534611 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524327    4447 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:19:34.534611 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524330    4447 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:19:34.534611 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524333    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:19:34.534611 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524336    4447 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:19:34.534611 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524339    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:19:34.534611 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524341    4447 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:19:34.534611 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524344    4447 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:19:34.534611 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524346    4447 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:19:34.535118 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524349    4447 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:19:34.535118 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524351    4447 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:19:34.535118 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524353    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:19:34.535118 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524357    4447 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:19:34.535118 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524360    4447 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:19:34.535118 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524362    4447 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:19:34.535118 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524364    4447 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:19:34.535118 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524367    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:19:34.535118 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524369    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:19:34.535118 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524371    4447 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:19:34.535118 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524374    4447 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:19:34.535118 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524376    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:19:34.535118 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524378    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:19:34.535118 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524381    4447 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:19:34.535118 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524384    4447 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:19:34.535118 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524386    4447 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:19:34.535118 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524389    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:19:34.535118 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524391    4447 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:19:34.535118 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524394    4447 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:19:34.535667 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524397    4447 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:19:34.535667 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524400    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:19:34.535667 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524403    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:19:34.535667 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524406    4447 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:19:34.535667 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524409    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:19:34.535667 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524411    4447 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:19:34.535667 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524414    4447 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:19:34.535667 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524417    4447 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:19:34.535667 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524421    4447 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:19:34.535667 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524424    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:19:34.535667 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524427    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:19:34.535667 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524429    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:19:34.535667 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524432    4447 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:19:34.535667 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524434    4447 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:19:34.535667 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524436    4447 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:19:34.535667 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524439    4447 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:19:34.535667 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524442    4447 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:19:34.535667 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524445    4447 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:19:34.536295 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524447    4447 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:19:34.536295 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524449    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:19:34.536295 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524451    4447 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:19:34.536295 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524454    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:19:34.536295 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524456    4447 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:19:34.536295 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524459    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:19:34.536295 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524461    4447 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:19:34.536295 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524463    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:19:34.536295 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524466    4447 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:19:34.536295 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524470    4447 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:19:34.536295 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524472    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:19:34.536295 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524475    4447 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:19:34.536295 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524477    4447 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:19:34.536295 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524480    4447 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:19:34.536295 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524482    4447 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:19:34.536295 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524484    4447 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:19:34.536295 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524487    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:19:34.536295 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524489    4447 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:19:34.536295 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524491    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:19:34.537254 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524494    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:19:34.537254 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524496    4447 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:19:34.537254 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524499    4447 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:19:34.537254 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524501    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:19:34.537254 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524503    4447 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:19:34.537254 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524506    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:19:34.537254 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524509    4447 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:19:34.537254 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524511    4447 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:19:34.537254 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524513    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:19:34.537254 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524516    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:19:34.537254 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524518    4447 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:19:34.537254 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524521    4447 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:19:34.537254 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524524    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:19:34.537254 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524526    4447 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:19:34.537254 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.524551    4447 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:19:34.537254 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524589    4447 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:19:34.537759 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524594    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:19:34.537759 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524597    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:19:34.537759 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524599    4447 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:19:34.537759 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524602    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:19:34.537759 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524604    4447 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:19:34.537759 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524607    4447 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:19:34.537759 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524610    4447 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:19:34.537759 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524615    4447 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:19:34.537759 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524617    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:19:34.537759 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524620    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:19:34.537759 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524623    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:19:34.537759 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524625    4447 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:19:34.537759 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524627    4447 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:19:34.537759 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524630    4447 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:19:34.537759 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524632    4447 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:19:34.537759 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524635    4447 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:19:34.537759 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524637    4447 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:19:34.537759 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524640    4447 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:19:34.537759 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524642    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:19:34.538386 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524644    4447 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:19:34.538386 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524647    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:19:34.538386 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524649    4447 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:19:34.538386 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524652    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:19:34.538386 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524654    4447 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:19:34.538386 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524657    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:19:34.538386 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524659    4447 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:19:34.538386 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524662    4447 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:19:34.538386 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524665    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:19:34.538386 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524667    4447 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:19:34.538386 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524670    4447 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:19:34.538386 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524672    4447 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:19:34.538386 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524675    4447 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:19:34.538386 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524677    4447 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:19:34.538386 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524679    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:19:34.538386 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524682    4447 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:19:34.538386 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524684    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:19:34.538386 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524686    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:19:34.538386 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524689    4447 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:19:34.539072 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524691    4447 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:19:34.539072 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524695    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:19:34.539072 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524697    4447 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:19:34.539072 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524700    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:19:34.539072 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524702    4447 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:19:34.539072 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524704    4447 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:19:34.539072 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524707    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:19:34.539072 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524709    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:19:34.539072 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524712    4447 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:19:34.539072 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524714    4447 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:19:34.539072 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524716    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:19:34.539072 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524719    4447 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:19:34.539072 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524721    4447 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:19:34.539072 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524724    4447 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:19:34.539072 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524726    4447 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:19:34.539072 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524728    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:19:34.539072 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524731    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:19:34.539072 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524733    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:19:34.539072 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524736    4447 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:19:34.539072 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524738    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:19:34.539570 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524741    4447 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:19:34.539570 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524744    4447 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:19:34.539570 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524747    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:19:34.539570 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524749    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:19:34.539570 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524751    4447 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:19:34.539570 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524754    4447 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:19:34.539570 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524756    4447 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:19:34.539570 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524758    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:19:34.539570 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524761    4447 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:19:34.539570 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524764    4447 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:19:34.539570 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524766    4447 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:19:34.539570 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524768    4447 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:19:34.539570 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524770    4447 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:19:34.539570 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524774    4447 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:19:34.539570 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524777    4447 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:19:34.539570 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524780    4447 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:19:34.539570 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524783    4447 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:19:34.539570 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524786    4447 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:19:34.539570 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524789    4447 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:19:34.539570 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524791    4447 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:19:34.540069 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524793    4447 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:19:34.540069 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524796    4447 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:19:34.540069 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524798    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:19:34.540069 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524801    4447 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:19:34.540069 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524804    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:19:34.540069 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524806    4447 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:19:34.540069 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524808    4447 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:19:34.540069 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524811    4447 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:19:34.540069 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524813    4447 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:19:34.540069 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524815    4447 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:19:34.540069 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524818    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:19:34.540069 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524821    4447 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:19:34.540069 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524823    4447 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:19:34.540069 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524827    4447 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:19:34.540069 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524829    4447 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:19:34.540069 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524832    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:19:34.540069 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524834    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:19:34.540069 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524837    4447 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:19:34.540069 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524839    4447 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:19:34.540069 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524841    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:19:34.540545 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524844    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:19:34.540545 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524846    4447 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:19:34.540545 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524848    4447 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:19:34.540545 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524851    4447 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:19:34.540545 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524853    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:19:34.540545 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524856    4447 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:19:34.540545 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.524859    4447 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:19:34.540545 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.530110    4447 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 03:19:34.540545 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.530127    4447 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 03:19:34.540545 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.530149    4447 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 03:19:34.540545 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.530153    4447 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 03:19:34.540545 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530217    4447 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:19:34.540545 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530222    4447 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:19:34.540545 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530224    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:19:34.540545 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530227    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:19:34.540545 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530230    4447 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:19:34.540545 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530233    4447 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:19:34.540545 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530235    4447 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:19:34.540545 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530238    4447 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:19:34.541080 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530240    4447 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:19:34.541080 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530243    4447 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:19:34.541080 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530245    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:19:34.541080 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530248    4447 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:19:34.541080 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530251    4447 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:19:34.541080 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530253    4447 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:19:34.541080 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530255    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:19:34.541080 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530258    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:19:34.541080 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530265    4447 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:19:34.541080 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530268    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:19:34.541080 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530271    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:19:34.541080 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530274    4447 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:19:34.541080 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530276    4447 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:19:34.541080 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530279    4447 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:19:34.541080 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530281    4447 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:19:34.541080 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530284    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:19:34.541080 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530286    4447 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:19:34.541080 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530289    4447 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:19:34.541080 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530292    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:19:34.541080 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530294    4447 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:19:34.541697 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530297    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:19:34.541697 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530300    4447 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:19:34.541697 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530302    4447 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:19:34.541697 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530305    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:19:34.541697 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530307    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:19:34.541697 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530310    4447 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:19:34.541697 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530312    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:19:34.541697 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530315    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:19:34.541697 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530317    4447 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:19:34.541697 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530320    4447 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:19:34.541697 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530323    4447 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:19:34.541697 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530325    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:19:34.541697 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530328    4447 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:19:34.541697 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530330    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:19:34.541697 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530333    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:19:34.541697 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530335    4447 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:19:34.541697 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530338    4447 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:19:34.541697 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530340    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:19:34.541697 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530343    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:19:34.541697 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530345    4447 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:19:34.542482 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530348    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:19:34.542482 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530350    4447 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:19:34.542482 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530352    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:19:34.542482 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530355    4447 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:19:34.542482 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530358    4447 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:19:34.542482 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530360    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:19:34.542482 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530363    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:19:34.542482 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530365    4447 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:19:34.542482 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530368    4447 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:19:34.542482 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530370    4447 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:19:34.542482 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530373    4447 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:19:34.542482 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530376    4447 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:19:34.542482 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530379    4447 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:19:34.542482 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530381    4447 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:19:34.542482 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530384    4447 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:19:34.542482 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530387    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:19:34.542482 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530389    4447 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:19:34.542482 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530392    4447 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:19:34.542482 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530394    4447 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:19:34.542482 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530397    4447 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:19:34.543244 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530399    4447 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:19:34.543244 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530402    4447 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:19:34.543244 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530404    4447 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:19:34.543244 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530407    4447 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:19:34.543244 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530409    4447 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:19:34.543244 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530412    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:19:34.543244 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530414    4447 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:19:34.543244 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530417    4447 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:19:34.543244 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530419    4447 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:19:34.543244 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530421    4447 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:19:34.543244 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530424    4447 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:19:34.543244 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530427    4447 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:19:34.543244 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530429    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:19:34.543244 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530432    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:19:34.543244 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530434    4447 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:19:34.543244 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530437    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:19:34.543244 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530441    4447 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:19:34.543244 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530444    4447 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:19:34.543244 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530448    4447 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:19:34.543692 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530451    4447 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:19:34.543692 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530456    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:19:34.543692 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530459    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:19:34.543692 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530462    4447 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:19:34.543692 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530465    4447 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:19:34.543692 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530467    4447 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:19:34.543692 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530470    4447 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:19:34.543692 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530472    4447 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:19:34.543692 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530474    4447 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:19:34.543692 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530477    4447 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:19:34.543692 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530480    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:19:34.543692 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530482    4447 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:19:34.543692 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530484    4447 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:19:34.543692 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530487    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:19:34.543692 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530490    4447 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:19:34.543692 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530493    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:19:34.543692 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530495    4447 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:19:34.543692 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530497    4447 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:19:34.543692 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530500    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:19:34.544233 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.530527    4447 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:19:34.544233 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530684    4447 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:19:34.544233 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530688    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:19:34.544233 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530691    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:19:34.544233 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530693    4447 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:19:34.544233 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530696    4447 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:19:34.544233 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530701    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:19:34.544233 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530704    4447 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:19:34.544233 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530707    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:19:34.544233 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530709    4447 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:19:34.544233 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530712    4447 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:19:34.544233 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530714    4447 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:19:34.544233 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530717    4447 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:19:34.544233 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530719    4447 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:19:34.544233 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530721    4447 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:19:34.544233 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530724    4447 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:19:34.544710 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530727    4447 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:19:34.544710 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530729    4447 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:19:34.544710 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530732    4447 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:19:34.544710 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530734    4447 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:19:34.544710 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530737    4447 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:19:34.544710 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530739    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:19:34.544710 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530741    4447 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:19:34.544710 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530744    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:19:34.544710 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530746    4447 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:19:34.544710 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530748    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:19:34.544710 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530751    4447 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:19:34.544710 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530753    4447 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:19:34.544710 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530756    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:19:34.544710 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530758    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:19:34.544710 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530760    4447 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:19:34.544710 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530763    4447 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:19:34.544710 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530765    4447 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:19:34.544710 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530768    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:19:34.544710 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530770    4447 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:19:34.544710 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530773    4447 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:19:34.545271 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530775    4447 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:19:34.545271 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530777    4447 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:19:34.545271 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530780    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:19:34.545271 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530784    4447 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:19:34.545271 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530788    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:19:34.545271 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530791    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:19:34.545271 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530793    4447 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:19:34.545271 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530796    4447 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:19:34.545271 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530799    4447 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:19:34.545271 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530801    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:19:34.545271 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530803    4447 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:19:34.545271 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530806    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:19:34.545271 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530808    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:19:34.545271 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530811    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:19:34.545271 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530814    4447 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:19:34.545271 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530818    4447 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:19:34.545271 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530822    4447 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:19:34.545271 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530824    4447 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:19:34.545271 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530827    4447 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:19:34.545823 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530829    4447 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:19:34.545823 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530832    4447 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:19:34.545823 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530834    4447 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:19:34.545823 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530837    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:19:34.545823 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530839    4447 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:19:34.545823 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530841    4447 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:19:34.545823 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530844    4447 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:19:34.545823 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530847    4447 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:19:34.545823 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530849    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:19:34.545823 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530851    4447 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:19:34.545823 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530854    4447 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:19:34.545823 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530856    4447 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:19:34.545823 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530859    4447 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:19:34.545823 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530862    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:19:34.545823 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530864    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:19:34.545823 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530866    4447 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:19:34.545823 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530869    4447 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:19:34.545823 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530871    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:19:34.545823 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530874    4447 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:19:34.545823 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530876    4447 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:19:34.546560 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530879    4447 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:19:34.546560 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530903    4447 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:19:34.546560 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530906    4447 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:19:34.546560 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530908    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:19:34.546560 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530911    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:19:34.546560 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530913    4447 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:19:34.546560 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530916    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:19:34.546560 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530919    4447 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:19:34.546560 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530921    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:19:34.546560 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530924    4447 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:19:34.546560 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530926    4447 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:19:34.546560 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530928    4447 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:19:34.546560 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530931    4447 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:19:34.546560 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530933    4447 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:19:34.546560 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530935    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:19:34.546560 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530938    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:19:34.546560 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530940    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:19:34.546560 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530943    4447 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:19:34.546560 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530945    4447 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:19:34.547144 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530947    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:19:34.547144 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530950    4447 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:19:34.547144 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530952    4447 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:19:34.547144 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530955    4447 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:19:34.547144 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530957    4447 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:19:34.547144 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530960    4447 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:19:34.547144 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530962    4447 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:19:34.547144 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530965    4447 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:19:34.547144 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530967    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:19:34.547144 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530969    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:19:34.547144 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530972    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:19:34.547144 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530975    4447 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:19:34.547144 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.530978    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:19:34.547144 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.531002    4447 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:19:34.547144 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531035    4447 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:19:34.547144 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531040    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:19:34.547605 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531043    4447 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:19:34.547605 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531046    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:19:34.547605 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531048    4447 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:19:34.547605 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531050    4447 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:19:34.547605 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531053    4447 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:19:34.547605 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531055    4447 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:19:34.547605 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531058    4447 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:19:34.547605 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531060    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:19:34.547605 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531062    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:19:34.547605 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531065    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:19:34.547605 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531067    4447 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:19:34.547605 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531069    4447 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:19:34.547605 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531072    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:19:34.547605 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531075    4447 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:19:34.547605 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531077    4447 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:19:34.547605 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531080    4447 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:19:34.547605 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531082    4447 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:19:34.547605 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531085    4447 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:19:34.547605 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531087    4447 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:19:34.548177 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531089    4447 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:19:34.548177 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531092    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:19:34.548177 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531094    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:19:34.548177 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531097    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:19:34.548177 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531099    4447 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:19:34.548177 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531101    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:19:34.548177 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531104    4447 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:19:34.548177 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531106    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:19:34.548177 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531109    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:19:34.548177 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531112    4447 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:19:34.548177 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531114    4447 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:19:34.548177 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531116    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:19:34.548177 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531119    4447 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:19:34.548177 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531121    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:19:34.548177 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531123    4447 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:19:34.548177 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531126    4447 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:19:34.548177 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531128    4447 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:19:34.548177 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531131    4447 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:19:34.548177 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531133    4447 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:19:34.548650 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531135    4447 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:19:34.548650 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531138    4447 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:19:34.548650 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531140    4447 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:19:34.548650 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531142    4447 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:19:34.548650 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531145    4447 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:19:34.548650 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531150    4447 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:19:34.548650 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531152    4447 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:19:34.548650 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531154    4447 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:19:34.548650 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531157    4447 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:19:34.548650 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531159    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:19:34.548650 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531161    4447 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:19:34.548650 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531164    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:19:34.548650 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531167    4447 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:19:34.548650 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531169    4447 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:19:34.548650 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531172    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:19:34.548650 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531174    4447 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:19:34.548650 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531176    4447 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:19:34.548650 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531179    4447 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:19:34.548650 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531181    4447 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:19:34.549170 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531184    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:19:34.549170 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531186    4447 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:19:34.549170 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531188    4447 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:19:34.549170 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531191    4447 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:19:34.549170 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531194    4447 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:19:34.549170 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531196    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:19:34.549170 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531200    4447 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:19:34.549170 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531203    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:19:34.549170 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531206    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:19:34.549170 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531209    4447 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:19:34.549170 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531212    4447 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:19:34.549170 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531214    4447 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:19:34.549170 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531217    4447 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:19:34.549170 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531219    4447 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:19:34.549170 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531222    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:19:34.549170 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531224    4447 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:19:34.549170 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531226    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:19:34.549170 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531230    4447 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:19:34.549170 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531235    4447 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:19:34.549794 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531238    4447 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:19:34.549794 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531241    4447 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:19:34.549794 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531243    4447 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:19:34.549794 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531246    4447 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:19:34.549794 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531248    4447 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:19:34.549794 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531250    4447 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:19:34.549794 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531253    4447 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:19:34.549794 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531255    4447 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:19:34.549794 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531258    4447 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:19:34.549794 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531260    4447 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:19:34.549794 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531262    4447 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:19:34.549794 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531265    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:19:34.549794 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531267    4447 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:19:34.549794 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531269    4447 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:19:34.549794 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531272    4447 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:19:34.549794 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531274    4447 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:19:34.549794 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531276    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:19:34.549794 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531279    4447 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:19:34.549794 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531282    4447 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:19:34.549794 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531284    4447 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:19:34.550637 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531286    4447 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:19:34.550637 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531289    4447 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:19:34.550637 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531291    4447 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:19:34.550637 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531294    4447 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:19:34.550637 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531296    4447 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:19:34.550637 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531300    4447 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:19:34.550637 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531303    4447 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:19:34.550637 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:34.531306    4447 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:19:34.550637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.531366    4447 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 03:19:34.550637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.532382    4447 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 03:19:34.550637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.532437    4447 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 03:19:34.550637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.532624    4447 server.go:1014] "Starting client certificate rotation"
Oct 31 03:19:34.550637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.532771    4447 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 03:19:34.550637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.532830    4447 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:13:55 +0000 UTC" deadline="2028-06-07 16:44:35.829346428 +0000 UTC"
Oct 31 03:19:34.550637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.532858    4447 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14053h25m1.29649054s"
Oct 31 03:19:34.550637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.533078    4447 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 03:19:34.551252 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.534226    4447 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 03:19:34.551252 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.535673    4447 log.go:25] "Validated CRI v1 runtime API"
Oct 31 03:19:34.551252 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.537624    4447 log.go:25] "Validated CRI v1 image API"
Oct 31 03:19:34.551252 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.538459    4447 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 03:19:34.551252 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.540448    4447 fs.go:135] Filesystem UUIDs: map[5da6bae3-0e3d-45a2-b121-c50b9b3927ca:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 93f112c0-26c5-439b-8707-25e3a4b2478c:/dev/nvme0n1p4]
Oct 31 03:19:34.551490 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.540472    4447 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/085b4d9646162e9856a356ee4e254e4a9a0aeca688bbdbc97d29c988a7c603cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/085b4d9646162e9856a356ee4e254e4a9a0aeca688bbdbc97d29c988a7c603cb/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/17e3b9117877b2bdb85084852dbe0e02a35b7a475587990611ffd6d46c70f4c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/17e3b9117877b2bdb85084852dbe0e02a35b7a475587990611ffd6d46c70f4c7/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ba34688772781b5f2a9085ffdcd167ec4f814dbc44c6f390f2e71d567c903b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ba34688772781b5f2a9085ffdcd167ec4f814dbc44c6f390f2e71d567c903b5/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ba1e5f242ab5ff991d731d0b55cbe7bc647054d69432e3870be8b507e1c5df1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ba1e5f242ab5ff991d731d0b55cbe7bc647054d69432e3870be8b507e1c5df1/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4195bb655ad65a47f514d13d6332248916eb5bb68e854977c1030d712ad41c51/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4195bb655ad65a47f514d13d6332248916eb5bb68e854977c1030d712ad41c51/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b306c03ddb8748a6e567db9c88ae13d9884f8f0e5b71fcadd5acc1bd96a1623/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b306c03ddb8748a6e567db9c88ae13d9884f8f0e5b71fcadd5acc1bd96a1623/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bba2411ce2baf8a44cbc6b4ccfbeb6de781689e24f146e060d96d64fdb1d5c18/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bba2411ce2baf8a44cbc6b4ccfbeb6de781689e24f146e060d96d64fdb1d5c18/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c65a43209eec3ae1f9aae6e32e4f0e7e8b6b7ca011b7a2f5b7964969ab0dd3a3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c65a43209eec3ae1f9aae6e32e4f0e7e8b6b7ca011b7a2f5b7964969ab0dd3a3/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d51312290a9c7031fe0fa5522d6e9e874f84f23859cda8a6a5e7dc0bff8e8fbc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d51312290a9c7031fe0fa5522d6e9e874f84f23859cda8a6a5e7dc0bff8e8fbc/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d61cdec76693cad5f1f6a9093e9d96d92957e4e6bb5f94af9ed9c4c5e51024a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d61cdec76693cad5f1f6a9093e9d96d92957e4e6bb5f94af9ed9c4c5e51024a2/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec067d0b6556a679cfccec988c00f2d6cc264e41aaae4cc591b7d6c03c5b8512/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec067d0b6556a679cfccec988c00f2d6cc264e41aaae4cc591b7d6c03c5b8512/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef4e18b14b5e5a470e8bd53b4789e825597d4221cfac7207bc810d3fbce498a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef4e18b14b5e5a470e8bd53b4789e825597d4221cfac7207bc810d3fbce498a8/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f42f3f365dbd3ecb6399cde9c9c656fe561a2142f8431d37660ad50902ff8a41/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f42f3f365dbd3ecb6399cde9c9c656fe561a2142f8431d37660ad50902ff8a41/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f80f3ff5d1cb2d8af42e4ca1df00bb5cd38c4aca3ac41c988f141e40f8c30ff7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f80f3ff5d1cb2d8af42e4ca1df00bb5cd38c4aca3ac41c988f141e40f8c30ff7/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2aa25e13-aaa4-4795-a221-2056b76e8c8f/volumes/kubernetes.io~projected/kube-api-access-6tpl7:{mountpoint:/var/lib/kubelet/pods/2aa25e13-aaa4-4795-a221-2056b76e8c8f/volumes/kubernetes.io~projected/kube-api-access-6tpl7 major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9/volumes/kubernetes.io~projected/kube-api-access-2hjs5:{mountpoint:/var/lib/kubelet/pods/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9/volumes/kubernetes.io~projected/kube-api-access-2hjs5 major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce/volumes/kubernetes.io~projected/kube-api-access-j92zc:{mountpoint:/var/lib/kubelet/pods/65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce/volumes/kubernetes.io~projected/kube-api-access-j92zc major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74face88-7a89-4b8f-9e8b-adea29761ed8/volumes/kubernetes.io~projected/kube-api-access-5zzzr:{mountpoint:/var/lib/kubelet/pods/74face88-7a89-4b8f-9e8b-adea29761ed8/volumes/kubernetes.io~projected/kube-api-access-5zzzr major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d72ff8d-4480-4266-ab8b-a225f4245836/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/7d72ff8d-4480-4266-ab8b-a225f4245836/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d72ff8d-4480-4266-ab8b-a225f4245836/volumes/kubernetes.io~projected/kube-api-access-jjkmq:{mountpoint:/var/lib/kubelet/pods/7d72ff8d-4480-4266-ab8b-a225f4245836/volumes/kubernetes.io~projected/kube-api-access-jjkmq major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d72ff8d-4480-4266-ab8b-a225f4245836/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/7d72ff8d-4480-4266-ab8b-a225f4245836/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d251d9b-c5e7-46e7-b5ab-524b8171d83b/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8d251d9b-c5e7-46e7-b5ab-524b8171d83b/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8da6593d-d284-438d-96ee-ec0d9ddf1124/volumes/kubernetes.io~projected/kube-api-access-s8ppp:{mountpoint:/var/lib/kubelet/pods/8da6593d-d284-438d-96ee-ec0d9ddf1124/volumes/kubernetes.io~projected/kube-api-access-s8ppp major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95bd2ef0-32a9-4150-abee-d2a9e03cc62c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/95bd2ef0-32a9-4150-abee-d2a9e03cc62c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95bd2ef0-32a9-4150-abee-d2a9e03cc62c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/95bd2ef0-32a9-4150-abee-d2a9e03cc62c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95bd2ef0-32a9-4150-abee-d2a9e03cc62c/volumes/kubernetes.io~projected/kube-api-access-n4wv2:{mountpoint:/var/lib/kubelet/pods/95bd2ef0-32a9-4150-abee-d2a9e03cc62c/volumes/kubernetes.io~projected/kube-api-access-n4wv2 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0f3be01-6bec-4a39-980a-e6342fcb3624/volumes/kubernetes.io~projected/kube-api-access-cjh4p:{mountpoint:/var/lib/kubelet/pods/a0f3be01-6bec-4a39-980a-e6342fcb3624/volumes/kubernetes.io~projected/kube-api-access-cjh4p major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0f3be01-6bec-4a39-980a-e6342fcb3624/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/a0f3be01-6bec-4a39-980a-e6342fcb3624/volumes/kubernetes.io~secret/signing-key major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7360ff6-d326-485b-bd3d-fd2f0f9099b5/volumes/kubernetes.io~projected/kube-api-access-kz5wr:{mountpoint:/var/lib/kubelet/pods/a7360ff6-d326-485b-bd3d-fd2f0f9099b5/volumes/kubernetes.io~projected/kube-api-access-kz5wr major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7360ff6-d326-485b-bd3d-fd2f0f9099b5/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a7360ff6-d326-485b-bd3d-fd2f0f9099b5/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac9dbe77-127f-4a88-9634-82560e6424ce/volumes/kubernetes.io~projected/kube-api-access-kxphc:{mountpoint:/var/lib/kubelet/pods/ac9dbe77-127f-4a88-9634-82560e6424ce/volumes/kubernetes.io~projected/kube-api-access-kxphc major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcb6233e-08f7-4a15-b273-b705bf9c2ac7/volumes/kubernetes.io~projected/kube-api-access-b2jf8:{mountpoint:/var/lib/kubelet/pods/bcb6233e-08f7-4a15-b273-b705bf9c2ac7/volumes/kubernetes.io~projected/kube-api-access-b2jf8 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be4c5472-30cf-447a-85fe-b90f5f8b022a/volumes/kubernetes.io~projected/kube-api-access-5ln55:{mountpoint:/var/lib/kubelet/pods/be4c5472-30cf-447a-85fe-b90f5f8b022a/volumes/kubernetes.io~projected/kube-api-access-5ln55 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8/volumes/kubernetes.io~projected/kube-api-access-p5jxx:{mountpoint:/var/lib/kubelet/pods/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8/volumes/kubernetes.io~projected/kube-api-access-p5jxx major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ebb8e08d-8243-492f-aef5-d3b6590d9f47/volumes/kubernetes.io~projected/kube-api-access-cp4mc:{mountpoint:/var/lib/kubelet/pods/ebb8e08d-8243-492f-aef5-d3b6590d9f47/volumes/kubernetes.io~projected/kube-api-access-cp4mc major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff0da67d-c75e-4487-ba6a-b85e04a496d9/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/ff0da67d-c75e-4487-ba6a-b85e04a496d9/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/2387f1a7242015a3d52521b1f136383a302cb36f4671256a761a31a111d0cf82/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/268df47ef56991c2f87d8a597b28a00f3cff3059cdb2032961acda5c160428f9/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-107:{mountpoint:/var/lib/containers/storage/overlay/7cba235f2b3ecf41c7673a94c1f7b0f25d9e860306729ca1b289ddd1681b633c/merged major:0 minor:107 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/9e16fbfb065c066f8d06274f9791670f24da2901155ae52c58637e8c1b2ad45a/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/d559edd2f7e911b5079da178358649c95e0da8bcd18c6f0fdde2d1e3003d1223/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-116:{mountpoint:/var/lib/containers/storage/overlay/efb07bacab11559463580bb9e9e40e5f39a26417d5cae9cb6122e6db29d953d9/merged major:0 minor:116 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/0db06c06c88c45d51d0b3f4a8018f1908ca4e03b2f1b9308bf9ad96dee44bcb6/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b3bc6599557ac723648f50a1aa6271924cc8e15d5f2866059c9735369f906be1/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/9c996cb46aef9a87be571f9d4d76cce1613d87fcb8cc6c4cbc8ae78964a8bea2/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/7bf2dac970046a58bc43c9e15d68efc6dc15342d85e0c95b06ecf47ea3cb4f0e/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/e9b3ed3fb88709363f2a43b7540d0985b187ea13caea00a2ea0a7875a0eea395/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/0c021dc4debe4122d918316ad7b4ecb4477507d7d71ee9aaf1e977f8d99afdb9/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/08b9714c9815f4635c377311310cd4d35690656bdca63fed7089243ebb2b302a/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/da392b037f76e14a82dfeca51b1025f6326a6ca7927ea2a6ee820251413b1edf/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/7dfaf3f59378b35007fce542b015ce4aaba64157cd067721e5b8f90a58c96ef4/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/3d9c424e10ccb90c5f133f3c8f35135df0ce1a0d721b1a57aa3d55587e52861e/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/397cd46b07bd58527274026c3ec46a0783bfbb4ec72124782e88b1ea3844deec/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/a722f9de5242487ab5c540a9778ca2e35564e206e8297b0c3f9c7ea0aad8c4c2/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/a96841c83a9c557b2cfd6d202e05d3eaf4871416793161adaa87691f5898ab27/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/1c194f5abedf088efb3c1841b607025dd6313731a9fd3a7d6946a440e70a5338/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/5c106b1aa3e79dd7bce3b392374cd88d3184ad33e83236c1596b671476b3d2ea/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/e5444cd5790edf31eb124f7b6e572def68638f716b1be1e0724c6e197568df91/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/93155d7abf5acf6291cb866c549fc45aca645d37735b54438b445856d03fab4e/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/c17ab92db81fc1ef0fac41f314ded6d69e9d980923cf20fd75d5a088f764d1b4/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/e9e23afd8f5ddc885b22ce891cfee28fb3d53db571bfeef338ea4c94676b71cf/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/4fd9c21afc9a0461f519d04b9d00f42756c89a21ac4d4c5948de01a4167037ee/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/efdf76799037f843e8481cbda89be8d5325b59f6c6b1761174e35a214c7c988a/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/ec72056773ef388ad9f53be062d3f567b632fe9dcbb4e200607371d71e1821c3/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/d14b7d86df3cf49ef3f3a4ba30b8e25cf6219aaedbd7e8e7d751e9891869374e/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/4c5570fd82a89d34b727f0d20eac391ae00693134a0c64a7a6e6d3e6383074c8/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/56f5793b58b572a9c9f13c31ec2e63739288941026157388860444903c99415a/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/34a05c1e1d52118f7aa81998afede823e27e55a8c0ae51555d938e230f01d067/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/da1852f5f8120433e43898e65cee7b3faf180cbde7072208c98a4a75934643b3/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/f1defbe8b5bf2c07a1c062246b29addf658bce282edfda1ca85f5c59d136d462/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/5df503175dd296a808bdee4d56de791828c84780fb8ce09d2db8e717ba524414/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/a88a0690fbe56c529b9ca0df58aed2cc069f432030d5715a55c4a22e4237daaf/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/ec14d289e386ebb8bae752658e64ad08fdedb14ee160d9eb33e93bb1763c521a/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/4c4d814616b8bdbc16e92d9a82e13b87fca36949a3685b3923f2a104a87ccf51/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/96ee73563e6da301db08f1e2d6f16892de9e412f0cfebb5c02b342fc63e86c4b/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 03:19:34.551709 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.548925    4447 manager.go:217] Machine: {Timestamp:2026-10-31 03:19:34.548118231 +0000 UTC m=+0.059100605 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200816 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec214fd7e717df96091d645188425bcf SystemUUID:ec214fd7-e717-df96-091d-645188425bcf BootID:9f8e5106-2c1b-490b-bc47-9d7a2a1370b8 Filesystems:[{Device:/var/lib/kubelet/pods/ebb8e08d-8243-492f-aef5-d3b6590d9f47/volumes/kubernetes.io~projected/kube-api-access-cp4mc DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d72ff8d-4480-4266-ab8b-a225f4245836/volumes/kubernetes.io~projected/kube-api-access-jjkmq DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-116 DeviceMajor:0 DeviceMinor:116 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/be4c5472-30cf-447a-85fe-b90f5f8b022a/volumes/kubernetes.io~projected/kube-api-access-5ln55 DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8da6593d-d284-438d-96ee-ec0d9ddf1124/volumes/kubernetes.io~projected/kube-api-access-s8ppp DeviceMajor:0 DeviceMinor:54 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/95bd2ef0-32a9-4150-abee-d2a9e03cc62c/volumes/kubernetes.io~projected/kube-api-access-n4wv2 DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d72ff8d-4480-4266-ab8b-a225f4245836/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f80f3ff5d1cb2d8af42e4ca1df00bb5cd38c4aca3ac41c988f141e40f8c30ff7/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a0f3be01-6bec-4a39-980a-e6342fcb3624/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95bd2ef0-32a9-4150-abee-d2a9e03cc62c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/74face88-7a89-4b8f-9e8b-adea29761ed8/volumes/kubernetes.io~projected/kube-api-access-5zzzr DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d251d9b-c5e7-46e7-b5ab-524b8171d83b/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:221 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9/volumes/kubernetes.io~projected/kube-api-access-2hjs5 DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a7360ff6-d326-485b-bd3d-fd2f0f9099b5/volumes/kubernetes.io~projected/kube-api-access-kz5wr DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/17e3b9117877b2bdb85084852dbe0e02a35b7a475587990611ffd6d46c70f4c7/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec067d0b6556a679cfccec988c00f2d6cc264e41aaae4cc591b7d6c03c5b8512/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f42f3f365dbd3ecb6399cde9c9c656fe561a2142f8431d37660ad50902ff8a41/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95bd2ef0-32a9-4150-abee-d2a9e03cc62c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:52 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ba34688772781b5f2a9085ffdcd167ec4f814dbc44c6f390f2e71d567c903b5/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef4e18b14b5e5a470e8bd53b4789e825597d4221cfac7207bc810d3fbce498a8/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ba1e5f242ab5ff991d731d0b55cbe7bc647054d69432e3870be8b507e1c5df1/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4195bb655ad65a47f514d13d6332248916eb5bb68e854977c1030d712ad41c51/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0f3be01-6bec-4a39-980a-e6342fcb3624/volumes/kubernetes.io~projected/kube-api-access-cjh4p DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bcb6233e-08f7-4a15-b273-b705bf9c2ac7/volumes/kubernetes.io~projected/kube-api-access-b2jf8 DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b306c03ddb8748a6e567db9c88ae13d9884f8f0e5b71fcadd5acc1bd96a1623/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ac9dbe77-127f-4a88-9634-82560e6424ce/volumes/kubernetes.io~projected/kube-api-access-kxphc DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff0da67d-c75e-4487-ba6a-b85e04a496d9/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d51312290a9c7031fe0fa5522d6e9e874f84f23859cda8a6a5e7dc0bff8e8fbc/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8/volumes/kubernetes.io~projected/kube-api-access-p5jxx DeviceMajor:0 DeviceMinor:226 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c65a43209eec3ae1f9aae6e32e4f0e7e8b6b7ca011b7a2f5b7964969ab0dd3a3/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-107 DeviceMajor:0 DeviceMinor:107 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce/volumes/kubernetes.io~projected/kube-api-access-j92zc DeviceMajor:0 DeviceMinor:53 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bba2411ce2baf8a44cbc6b4ccfbeb6de781689e24f146e060d96d64fdb1d5c18/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a7360ff6-d326-485b-bd3d-fd2f0f9099b5/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d72ff8d-4480-4266-ab8b-a225f4245836/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:51 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2aa25e13-aaa4-4795-a221-2056b76e8c8f/volumes/kubernetes.io~projected/kube-api-access-6tpl7 DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/085b4d9646162e9856a356ee4e254e4a9a0aeca688bbdbc97d29c988a7c603cb/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d61cdec76693cad5f1f6a9093e9d96d92957e4e6bb5f94af9ed9c4c5e51024a2/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:17e3b9117877b2b MacAddress:d6:43:44:df:18:ea Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:69:28:50:19:8f Speed:0 Mtu:9001} {Name:br-int MacAddress:d6:32:5b:68:37:6f Speed:0 Mtu:8901} {Name:d61cdec76693cad MacAddress:f2:04:0d:9a:b8:5b Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:69:28:50:19:8f Speed:0 Mtu:9001} {Name:f42f3f365dbd3ec MacAddress:96:6d:c5:c4:ca:2b Speed:10000 Mtu:8901} {Name:f80f3ff5d1cb2d8 MacAddress:1e:86:a6:f5:53:5d Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:7e:95:68:cf:97:a5 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:b2:03:47:d5:ea:bf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 03:19:34.551709 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.549305    4447 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 03:19:34.551709 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.549362    4447 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 03:19:34.551709 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.549747    4447 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 03:19:34.551709 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.549781    4447 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-255.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 03:19:34.551709 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.549999    4447 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 03:19:34.551709 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.550011    4447 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 03:19:34.551709 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.550027    4447 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 03:19:34.551709 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.550050    4447 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 03:19:34.551709 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.550167    4447 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 03:19:34.551709 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.550409    4447 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 03:19:34.551709 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.550558    4447 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 03:19:34.551709 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.550611    4447 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 03:19:34.551709 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.550626    4447 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 03:19:34.551709 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.550643    4447 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 03:19:34.551709 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.550655    4447 kubelet.go:405] "Adding apiserver pod source"
Oct 31 03:19:34.551709 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.550666    4447 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 03:19:34.551709 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.551435    4447 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 03:19:34.551709 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.551447    4447 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 03:19:34.552427 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.552091    4447 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 03:19:34.554584 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.554567    4447 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 03:19:34.554669 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.554584    4447 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 03:19:34.554726 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.554715    4447 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 03:19:34.555053 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.555040    4447 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 03:19:34.555098 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.555088    4447 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 03:19:34.555209 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.555200    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 03:19:34.555247 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.555219    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 03:19:34.555247 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.555229    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 03:19:34.555247 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.555237    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 03:19:34.555247 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.555246    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 03:19:34.555350 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.555256    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 03:19:34.555350 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.555264    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 03:19:34.555350 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.555274    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 03:19:34.555350 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.555284    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 03:19:34.555350 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.555292    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 03:19:34.555350 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.555305    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 03:19:34.555350 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.555319    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 03:19:34.555350 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.555351    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 03:19:34.555728 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.555360    4447 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 03:19:34.558993 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.558978    4447 server.go:1263] "Started kubelet"
Oct 31 03:19:34.559099 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.559068    4447 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 03:19:34.559956 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.559213    4447 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 03:19:34.559956 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.559317    4447 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 03:19:34.559956 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.559615    4447 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 03:19:34.559537 ip-10-0-134-255 systemd[1]: Started Kubernetes Kubelet.
Oct 31 03:19:34.562303 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.562206    4447 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 03:19:34.562773 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.562715    4447 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 03:19:34.563304 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.563226    4447 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 03:19:34.563547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.563489    4447 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:13:56 +0000 UTC" deadline="2028-04-13 10:10:04.533367653 +0000 UTC"
Oct 31 03:19:34.563547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.563525    4447 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12726h50m29.969845694s"
Oct 31 03:19:34.563730 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.563699    4447 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 03:19:34.563730 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.563712    4447 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 03:19:34.563831 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.563827    4447 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 03:19:34.563909 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:19:34.563863    4447 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-255.us-west-1.compute.internal\" not found"
Oct 31 03:19:34.565871 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.565820    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac9dbe77-127f-4a88-9634-82560e6424ce" volumeName="kubernetes.io/projected/ac9dbe77-127f-4a88-9634-82560e6424ce-kube-api-access-kxphc" seLinuxMountContext=""
Oct 31 03:19:34.565871 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.565876    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9" volumeName="kubernetes.io/projected/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-kube-api-access-2hjs5" seLinuxMountContext=""
Oct 31 03:19:34.566082 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.565922    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d72ff8d-4480-4266-ab8b-a225f4245836" volumeName="kubernetes.io/projected/7d72ff8d-4480-4266-ab8b-a225f4245836-kube-api-access-jjkmq" seLinuxMountContext=""
Oct 31 03:19:34.566082 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.565937    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be4c5472-30cf-447a-85fe-b90f5f8b022a" volumeName="kubernetes.io/projected/be4c5472-30cf-447a-85fe-b90f5f8b022a-kube-api-access-5ln55" seLinuxMountContext=""
Oct 31 03:19:34.566082 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.565954    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff0da67d-c75e-4487-ba6a-b85e04a496d9" volumeName="kubernetes.io/configmap/ff0da67d-c75e-4487-ba6a-b85e04a496d9-konnectivity-ca" seLinuxMountContext=""
Oct 31 03:19:34.566082 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.565968    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2aa25e13-aaa4-4795-a221-2056b76e8c8f" volumeName="kubernetes.io/configmap/2aa25e13-aaa4-4795-a221-2056b76e8c8f-iptables-alerter-script" seLinuxMountContext=""
Oct 31 03:19:34.566082 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.565983    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0f3be01-6bec-4a39-980a-e6342fcb3624" volumeName="kubernetes.io/secret/a0f3be01-6bec-4a39-980a-e6342fcb3624-signing-key" seLinuxMountContext=""
Oct 31 03:19:34.566082 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.565999    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7360ff6-d326-485b-bd3d-fd2f0f9099b5" volumeName="kubernetes.io/projected/a7360ff6-d326-485b-bd3d-fd2f0f9099b5-kube-api-access-kz5wr" seLinuxMountContext=""
Oct 31 03:19:34.566082 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566014    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be4c5472-30cf-447a-85fe-b90f5f8b022a" volumeName="kubernetes.io/configmap/be4c5472-30cf-447a-85fe-b90f5f8b022a-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 31 03:19:34.566082 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566032    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce" volumeName="kubernetes.io/empty-dir/65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce-tmp-dir" seLinuxMountContext=""
Oct 31 03:19:34.566082 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566049    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74face88-7a89-4b8f-9e8b-adea29761ed8" volumeName="kubernetes.io/configmap/74face88-7a89-4b8f-9e8b-adea29761ed8-cni-binary-copy" seLinuxMountContext=""
Oct 31 03:19:34.566082 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566066    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d72ff8d-4480-4266-ab8b-a225f4245836" volumeName="kubernetes.io/configmap/7d72ff8d-4480-4266-ab8b-a225f4245836-ovnkube-config" seLinuxMountContext=""
Oct 31 03:19:34.566082 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566078    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8da6593d-d284-438d-96ee-ec0d9ddf1124" volumeName="kubernetes.io/configmap/8da6593d-d284-438d-96ee-ec0d9ddf1124-serviceca" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566089    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcb6233e-08f7-4a15-b273-b705bf9c2ac7" volumeName="kubernetes.io/projected/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-kube-api-access-b2jf8" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566109    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be4c5472-30cf-447a-85fe-b90f5f8b022a" volumeName="kubernetes.io/empty-dir/be4c5472-30cf-447a-85fe-b90f5f8b022a-data-volume" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566121    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2aa25e13-aaa4-4795-a221-2056b76e8c8f" volumeName="kubernetes.io/projected/2aa25e13-aaa4-4795-a221-2056b76e8c8f-kube-api-access-6tpl7" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566144    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcb6233e-08f7-4a15-b273-b705bf9c2ac7" volumeName="kubernetes.io/configmap/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566156    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0f3be01-6bec-4a39-980a-e6342fcb3624" volumeName="kubernetes.io/projected/a0f3be01-6bec-4a39-980a-e6342fcb3624-kube-api-access-cjh4p" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566172    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d72ff8d-4480-4266-ab8b-a225f4245836" volumeName="kubernetes.io/secret/7d72ff8d-4480-4266-ab8b-a225f4245836-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566183    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce" volumeName="kubernetes.io/projected/65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce-kube-api-access-j92zc" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566196    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74face88-7a89-4b8f-9e8b-adea29761ed8" volumeName="kubernetes.io/projected/74face88-7a89-4b8f-9e8b-adea29761ed8-kube-api-access-5zzzr" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566214    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d72ff8d-4480-4266-ab8b-a225f4245836" volumeName="kubernetes.io/configmap/7d72ff8d-4480-4266-ab8b-a225f4245836-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566226    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95bd2ef0-32a9-4150-abee-d2a9e03cc62c" volumeName="kubernetes.io/empty-dir/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-tmp" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566243    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95bd2ef0-32a9-4150-abee-d2a9e03cc62c" volumeName="kubernetes.io/projected/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-kube-api-access-n4wv2" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566255    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcb6233e-08f7-4a15-b273-b705bf9c2ac7" volumeName="kubernetes.io/configmap/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-cni-binary-copy" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566268    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8da6593d-d284-438d-96ee-ec0d9ddf1124" volumeName="kubernetes.io/projected/8da6593d-d284-438d-96ee-ec0d9ddf1124-kube-api-access-s8ppp" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566284    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0f3be01-6bec-4a39-980a-e6342fcb3624" volumeName="kubernetes.io/configmap/a0f3be01-6bec-4a39-980a-e6342fcb3624-signing-cabundle" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566299    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac9dbe77-127f-4a88-9634-82560e6424ce" volumeName="kubernetes.io/empty-dir/ac9dbe77-127f-4a88-9634-82560e6424ce-tmp-dir" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566324    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcb6233e-08f7-4a15-b273-b705bf9c2ac7" volumeName="kubernetes.io/configmap/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566336    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebb8e08d-8243-492f-aef5-d3b6590d9f47" volumeName="kubernetes.io/projected/ebb8e08d-8243-492f-aef5-d3b6590d9f47-kube-api-access-cp4mc" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566352    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d251d9b-c5e7-46e7-b5ab-524b8171d83b" volumeName="kubernetes.io/secret/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-original-pull-secret" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566364    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95bd2ef0-32a9-4150-abee-d2a9e03cc62c" volumeName="kubernetes.io/empty-dir/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-tuned" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566384    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7360ff6-d326-485b-bd3d-fd2f0f9099b5" volumeName="kubernetes.io/secret/a7360ff6-d326-485b-bd3d-fd2f0f9099b5-cert" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566398    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac9dbe77-127f-4a88-9634-82560e6424ce" volumeName="kubernetes.io/configmap/ac9dbe77-127f-4a88-9634-82560e6424ce-config-volume" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566410    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf1633f9-bdb5-4a26-bd80-a83a4b5587e8" volumeName="kubernetes.io/projected/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8-kube-api-access-p5jxx" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566426    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff0da67d-c75e-4487-ba6a-b85e04a496d9" volumeName="kubernetes.io/secret/ff0da67d-c75e-4487-ba6a-b85e04a496d9-agent-certs" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566439    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74face88-7a89-4b8f-9e8b-adea29761ed8" volumeName="kubernetes.io/configmap/74face88-7a89-4b8f-9e8b-adea29761ed8-multus-daemon-config" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566457    4447 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d72ff8d-4480-4266-ab8b-a225f4245836" volumeName="kubernetes.io/configmap/7d72ff8d-4480-4266-ab8b-a225f4245836-env-overrides" seLinuxMountContext=""
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566469    4447 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 03:19:34.566697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.566477    4447 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 03:19:34.571349 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.571333    4447 factory.go:153] Registering CRI-O factory
Oct 31 03:19:34.571430 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.571354    4447 factory.go:223] Registration of the crio container factory successfully
Oct 31 03:19:34.571430 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.571410    4447 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 03:19:34.571430 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.571430    4447 factory.go:55] Registering systemd factory
Oct 31 03:19:34.571575 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.571436    4447 factory.go:223] Registration of the systemd container factory successfully
Oct 31 03:19:34.571575 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.571458    4447 factory.go:103] Registering Raw factory
Oct 31 03:19:34.571575 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.571479    4447 manager.go:1199] Started watching for new ooms in manager
Oct 31 03:19:34.571853 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.571825    4447 manager.go:319] Starting recovery of all containers
Oct 31 03:19:34.587369 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.587346    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 03:19:34.587855 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.587837    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 03:19:34.597452 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.597404    4447 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 03:19:34.600012 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.599989    4447 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 03:19:34.600125 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.600016    4447 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 03:19:34.600125 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.600042    4447 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 03:19:34.600125 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:19:34.600095    4447 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 03:19:34.616562 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.616540    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 03:19:34.616914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.616870    4447 generic.go:356] "Generic (PLEG): container finished" podID="7d72ff8d-4480-4266-ab8b-a225f4245836" containerID="ccc19a43a88dce5acb62ea67d353bc533a7767d601c403d0d8a4e3a6f6ffd42d" exitCode=1
Oct 31 03:19:34.622437 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.622412    4447 generic.go:356] "Generic (PLEG): container finished" podID="2075c7dc615e009d2cc914067d053547" containerID="88711d43e6441034cbc31cc514ab04d40f74f4e6902efd78b50795b3e7ea72f5" exitCode=0
Oct 31 03:19:34.635832 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.633107    4447 generic.go:356] "Generic (PLEG): container finished" podID="bcb6233e-08f7-4a15-b273-b705bf9c2ac7" containerID="ab8d866ba94cbf183fad65bef9d24de7bc51efd5ab783c004d46eba3a58a0740" exitCode=0
Oct 31 03:19:34.635832 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.633279    4447 generic.go:356] "Generic (PLEG): container finished" podID="bcb6233e-08f7-4a15-b273-b705bf9c2ac7" containerID="a75232eea5bae4aee8d8e20d8741e2dbd8f852258e4936b5d9b3a1e1b23c0823" exitCode=0
Oct 31 03:19:34.635832 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.633292    4447 generic.go:356] "Generic (PLEG): container finished" podID="bcb6233e-08f7-4a15-b273-b705bf9c2ac7" containerID="6fb7c479af725e661f4e4368c7dbdaf1e8885810c6906443a8d958e6a39fe475" exitCode=0
Oct 31 03:19:34.635832 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.633300    4447 generic.go:356] "Generic (PLEG): container finished" podID="bcb6233e-08f7-4a15-b273-b705bf9c2ac7" containerID="bc234353a20d91ee34a30ffc06fdc8903a0797604e3b25e10fdce212de1fea20" exitCode=0
Oct 31 03:19:34.635832 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.633309    4447 generic.go:356] "Generic (PLEG): container finished" podID="bcb6233e-08f7-4a15-b273-b705bf9c2ac7" containerID="3dd64bb5386f6ecef726aae51ade85cfef7dca5a50665271b6e3e71830b0cdb1" exitCode=0
Oct 31 03:19:34.635832 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.633317    4447 generic.go:356] "Generic (PLEG): container finished" podID="bcb6233e-08f7-4a15-b273-b705bf9c2ac7" containerID="005714943d9331bce0ebde8ebf14dc484731ba562309490cb77c206e989413a4" exitCode=0
Oct 31 03:19:34.636283 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.635997    4447 manager.go:324] Recovery completed
Oct 31 03:19:34.648621 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.648603    4447 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:19:34.649800 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.649781    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:19:34.649901 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.649811    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:19:34.649901 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.649822    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:19:34.651157 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.651141    4447 cpu_manager.go:226] "Starting" policy="none"
Oct 31 03:19:34.651157 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.651153    4447 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 03:19:34.651282 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.651170    4447 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 03:19:34.651282 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.651151    4447 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:19:34.651374 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.651284    4447 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 03:19:34.651374 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.651294    4447 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 03:19:34.651374 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.651322    4447 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 03:19:34.651374 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.651327    4447 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 03:19:34.651374 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.651333    4447 policy_none.go:50] "Start"
Oct 31 03:19:34.651374 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.651347    4447 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 03:19:34.651374 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.651355    4447 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 03:19:34.651552 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.651505    4447 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 03:19:34.651552 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.651513    4447 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 03:19:34.651552 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.651518    4447 policy_none.go:44] "Start"
Oct 31 03:19:34.652029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.652014    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:19:34.652111 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.652046    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:19:34.652111 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.652060    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:19:34.655452 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.655387    4447 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 03:19:34.658463 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.658449    4447 manager.go:341] "Starting Device Plugin manager"
Oct 31 03:19:34.658550 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:19:34.658478    4447 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 03:19:34.658550 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.658490    4447 server.go:87] "Starting device plugin registration server"
Oct 31 03:19:34.658793 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.658780    4447 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 03:19:34.658859 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.658796    4447 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 03:19:34.658925 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.658912    4447 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 03:19:34.659746 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.659702    4447 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 03:19:34.659831 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.659757    4447 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 03:19:34.659831 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.659764    4447 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 03:19:34.659962 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.659838    4447 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T03:19:34.659743076Z","UUID":"c628b17c-fd53-4bc1-9a90-3c701420f603","Handler":null,"Name":"","Endpoint":""}
Oct 31 03:19:34.660301 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:19:34.660269    4447 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 03:19:34.660382 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:19:34.660307    4447 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-255.us-west-1.compute.internal\" not found"
Oct 31 03:19:34.661330 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.661313    4447 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 03:19:34.661432 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.661417    4447 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 03:19:34.701066 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.700504    4447 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal"]
Oct 31 03:19:34.701220 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.701151    4447 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:19:34.702084 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.702070    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:19:34.702153 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.702098    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:19:34.702153 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.702107    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:19:34.702353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.702343    4447 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:19:34.702517 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.702505    4447 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:19:34.703130 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.703112    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:19:34.703222 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.703144    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:19:34.703222 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.703153    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:19:34.703222 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.703115    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:19:34.703222 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.703212    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:19:34.703222 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.703223    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:19:34.703369 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:19:34.703291    4447 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-255.us-west-1.compute.internal\" not found" node="ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:19:34.703468 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.703441    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal" event={"ID":"2075c7dc615e009d2cc914067d053547","Type":"ContainerStarted","Data":"cedb4e29fc2f98c29b526d5519c012c78bd0cc60d2e1393ff7d12ff7e4443f83"}
Oct 31 03:19:34.703502 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.703483    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal" event={"ID":"2075c7dc615e009d2cc914067d053547","Type":"ContainerDied","Data":"88711d43e6441034cbc31cc514ab04d40f74f4e6902efd78b50795b3e7ea72f5"}
Oct 31 03:19:34.703502 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.703498    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal" event={"ID":"2075c7dc615e009d2cc914067d053547","Type":"ContainerStarted","Data":"6b306c03ddb8748a6e567db9c88ae13d9884f8f0e5b71fcadd5acc1bd96a1623"}
Oct 31 03:19:34.703558 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.703510    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal" event={"ID":"d21c869370d277ce1b45bad9863021dc","Type":"ContainerStarted","Data":"9ae1bca4c6062418d01e42e9feccd0a77c0108459338f83cb0271395dc707349"}
Oct 31 03:19:34.703558 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.703517    4447 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:19:34.703558 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.703522    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal" event={"ID":"d21c869370d277ce1b45bad9863021dc","Type":"ContainerStarted","Data":"1ba34688772781b5f2a9085ffdcd167ec4f814dbc44c6f390f2e71d567c903b5"}
Oct 31 03:19:34.704189 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.704178    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:19:34.704233 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.704199    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:19:34.704233 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.704209    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:19:34.704339 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:19:34.704326    4447 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-255.us-west-1.compute.internal\" not found" node="ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:19:34.759480 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.759451    4447 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:19:34.760523 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.760507    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:19:34.760601 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.760535    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:19:34.760601 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.760545    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:19:34.760601 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.760564    4447 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:19:34.761578 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.761556    4447 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 03:19:34.762573 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.762549    4447 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:19:34.762683 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.762651    4447 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:19:34.762758 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.762743    4447 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:19:34.765582 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.765560    4447 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:19:34.812608 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.812580    4447 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:19:34.860227 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.860207    4447 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:19:34.860346 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:19:34.860257    4447 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:19:34.860346 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.860272    4447 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:19:34.862413 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.862394    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2075c7dc615e009d2cc914067d053547-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal\" (UID: \"2075c7dc615e009d2cc914067d053547\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:19:34.862463 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.862422    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2075c7dc615e009d2cc914067d053547-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal\" (UID: \"2075c7dc615e009d2cc914067d053547\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:19:34.862502 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.862465    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d21c869370d277ce1b45bad9863021dc-config\") pod \"kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal\" (UID: \"d21c869370d277ce1b45bad9863021dc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:19:34.913510 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.913448    4447 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:19:34.913591 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.913517    4447 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:19:34.914121 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.914102    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeNotReady"
Oct 31 03:19:34.914244 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.914138    4447 setters.go:546] "Node became not ready" node="ip-10-0-134-255.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-31T03:19:34Z","lastTransitionTime":"2026-10-31T03:19:34Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 31 03:19:34.924616 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.924592    4447 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:19:34.924701 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:19:34.924641    4447 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:19:34.939113 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.939087    4447 nodeinfomanager.go:471] existing CSINode "ip-10-0-134-255.us-west-1.compute.internal" is owned by different node (oldNodeID="1ebb5296-25ff-44d3-8eef-7b589d9c8200", newNodeID=""), cleaning up...
Oct 31 03:19:34.963342 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.963310    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2075c7dc615e009d2cc914067d053547-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal\" (UID: \"2075c7dc615e009d2cc914067d053547\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:19:34.963462 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.963329    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2075c7dc615e009d2cc914067d053547-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal\" (UID: \"2075c7dc615e009d2cc914067d053547\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:19:34.963462 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.963382    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2075c7dc615e009d2cc914067d053547-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal\" (UID: \"2075c7dc615e009d2cc914067d053547\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:19:34.963462 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.963407    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d21c869370d277ce1b45bad9863021dc-config\") pod \"kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal\" (UID: \"d21c869370d277ce1b45bad9863021dc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:19:34.963462 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.963443    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2075c7dc615e009d2cc914067d053547-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal\" (UID: \"2075c7dc615e009d2cc914067d053547\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:19:34.963462 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:34.963458    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d21c869370d277ce1b45bad9863021dc-config\") pod \"kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal\" (UID: \"d21c869370d277ce1b45bad9863021dc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:19:35.551665 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.551640    4447 apiserver.go:52] "Watching apiserver"
Oct 31 03:19:35.639936 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.639907    4447 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:19:35.640067 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.640042    4447 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:19:35.655627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.655606    4447 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 03:19:35.656050 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.656032    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-j86qh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal","openshift-network-diagnostics/network-check-target-rtp6w","kube-system/kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal","kube-system/konnectivity-agent-kqgxt","openshift-service-ca/service-ca-5fffc97bdf-tqfr6","openshift-ingress-canary/ingress-canary-r9prp","openshift-multus/multus-additional-cni-plugins-lkj57","openshift-multus/network-metrics-daemon-bx98q","openshift-image-registry/node-ca-2kzdk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp","openshift-dns/dns-default-zxbcc","openshift-cluster-node-tuning-operator/tuned-l494z","kube-system/global-pull-secret-syncer-7nb9d","openshift-network-operator/iptables-alerter-9x2kw","openshift-insights/insights-runtime-extractor-qbl5h","openshift-ovn-kubernetes/ovnkube-node-rkkr4","openshift-multus/multus-vxkwg"]
Oct 31 03:19:35.656455 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.656442    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:19:35.663566 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.663543    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:35.663962 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.663942    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:35.668160 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668137    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-sysconfig\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.668160 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668162    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-kubernetes\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.668302 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668178    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-system-cni-dir\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.668302 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668197    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/74face88-7a89-4b8f-9e8b-adea29761ed8-multus-daemon-config\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.668302 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668219    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz5wr\" (UniqueName: \"kubernetes.io/projected/a7360ff6-d326-485b-bd3d-fd2f0f9099b5-kube-api-access-kz5wr\") pod \"ingress-canary-r9prp\" (UID: \"a7360ff6-d326-485b-bd3d-fd2f0f9099b5\") " pod="openshift-ingress-canary/ingress-canary-r9prp"
Oct 31 03:19:35.668302 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668271    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-sys-fs\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:19:35.668427 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668324    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-var-lib-kubelet\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.668427 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668353    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-cnibin\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.668427 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668367    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-var-lib-kubelet\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.668427 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668385    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac9dbe77-127f-4a88-9634-82560e6424ce-tmp-dir\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:35.668536 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668433    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5jxx\" (UniqueName: \"kubernetes.io/projected/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8-kube-api-access-p5jxx\") pod \"network-check-target-rtp6w\" (UID: \"bf1633f9-bdb5-4a26-bd80-a83a4b5587e8\") " pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:19:35.668536 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668462    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac9dbe77-127f-4a88-9634-82560e6424ce-tmp-dir\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:35.668536 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668496    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-slash\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.668536 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668524    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-run-netns\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.668641 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668540    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:19:35.668641 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668593    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-systemd\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.668641 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668618    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-run-multus-certs\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.668723 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668646    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7360ff6-d326-485b-bd3d-fd2f0f9099b5-cert\") pod \"ingress-canary-r9prp\" (UID: \"a7360ff6-d326-485b-bd3d-fd2f0f9099b5\") " pod="openshift-ingress-canary/ingress-canary-r9prp"
Oct 31 03:19:35.668723 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668682    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce-hosts-file\") pod \"node-resolver-j86qh\" (UID: \"65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce\") " pod="openshift-dns/node-resolver-j86qh"
Oct 31 03:19:35.668723 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668703    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-cni-netd\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.668803 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668737    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/be4c5472-30cf-447a-85fe-b90f5f8b022a-data-volume\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:35.668803 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668754    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp4mc\" (UniqueName: \"kubernetes.io/projected/ebb8e08d-8243-492f-aef5-d3b6590d9f47-kube-api-access-cp4mc\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:19:35.668803 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668770    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.668803 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668787    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:19:35.668936 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668802    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2jf8\" (UniqueName: \"kubernetes.io/projected/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-kube-api-access-b2jf8\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:19:35.668936 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668836    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs\") pod \"network-metrics-daemon-bx98q\" (UID: \"45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9\") " pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:19:35.668936 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668867    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ff0da67d-c75e-4487-ba6a-b85e04a496d9-agent-certs\") pod \"konnectivity-agent-kqgxt\" (UID: \"ff0da67d-c75e-4487-ba6a-b85e04a496d9\") " pod="kube-system/konnectivity-agent-kqgxt"
Oct 31 03:19:35.668936 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668907    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ff0da67d-c75e-4487-ba6a-b85e04a496d9-konnectivity-ca\") pod \"konnectivity-agent-kqgxt\" (UID: \"ff0da67d-c75e-4487-ba6a-b85e04a496d9\") " pod="kube-system/konnectivity-agent-kqgxt"
Oct 31 03:19:35.668936 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668920    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/be4c5472-30cf-447a-85fe-b90f5f8b022a-data-volume\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:35.669100 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668942    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-registration-dir\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:19:35.669100 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.668966    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-etc-selinux\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:19:35.669100 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669009    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-multus-conf-dir\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.669100 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669051    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxphc\" (UniqueName: \"kubernetes.io/projected/ac9dbe77-127f-4a88-9634-82560e6424ce-kube-api-access-kxphc\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:35.669100 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669075    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-kubelet\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.669259 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669103    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-run-systemd\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.669259 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669154    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-var-lib-openvswitch\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.669259 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669186    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-cni-binary-copy\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:19:35.669259 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669210    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2aa25e13-aaa4-4795-a221-2056b76e8c8f-iptables-alerter-script\") pod \"iptables-alerter-9x2kw\" (UID: \"2aa25e13-aaa4-4795-a221-2056b76e8c8f\") " pod="openshift-network-operator/iptables-alerter-9x2kw"
Oct 31 03:19:35.669259 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669246    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-etc-kubernetes\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.669500 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669261    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-run-ovn\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.669500 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669276    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-node-log\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.669500 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669308    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-cni-bin\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.669500 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669327    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:35.669500 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669342    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-hostroot\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.669500 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669360    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-dbus\") pod \"global-pull-secret-syncer-7nb9d\" (UID: \"8d251d9b-c5e7-46e7-b5ab-524b8171d83b\") " pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:35.669500 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669376    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-host\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.669500 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669405    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2aa25e13-aaa4-4795-a221-2056b76e8c8f-host-slash\") pod \"iptables-alerter-9x2kw\" (UID: \"2aa25e13-aaa4-4795-a221-2056b76e8c8f\") " pod="openshift-network-operator/iptables-alerter-9x2kw"
Oct 31 03:19:35.669500 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669426    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-sysctl-conf\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.669500 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669446    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j92zc\" (UniqueName: \"kubernetes.io/projected/65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce-kube-api-access-j92zc\") pod \"node-resolver-j86qh\" (UID: \"65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce\") " pod="openshift-dns/node-resolver-j86qh"
Oct 31 03:19:35.669751 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669548    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tpl7\" (UniqueName: \"kubernetes.io/projected/2aa25e13-aaa4-4795-a221-2056b76e8c8f-kube-api-access-6tpl7\") pod \"iptables-alerter-9x2kw\" (UID: \"2aa25e13-aaa4-4795-a221-2056b76e8c8f\") " pod="openshift-network-operator/iptables-alerter-9x2kw"
Oct 31 03:19:35.669751 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669567    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-var-lib-cni-bin\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.669751 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669583    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce-tmp-dir\") pod \"node-resolver-j86qh\" (UID: \"65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce\") " pod="openshift-dns/node-resolver-j86qh"
Oct 31 03:19:35.669751 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669601    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-run-openvswitch\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.669751 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669634    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-log-socket\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.669751 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669687    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce-tmp-dir\") pod \"node-resolver-j86qh\" (UID: \"65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce\") " pod="openshift-dns/node-resolver-j86qh"
Oct 31 03:19:35.669751 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669692    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d72ff8d-4480-4266-ab8b-a225f4245836-ovnkube-config\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.669751 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669728    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ln55\" (UniqueName: \"kubernetes.io/projected/be4c5472-30cf-447a-85fe-b90f5f8b022a-kube-api-access-5ln55\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:35.669982 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669767    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-device-dir\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:19:35.669982 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669785    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-etc-openvswitch\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.669982 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669857    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-original-pull-secret\") pod \"global-pull-secret-syncer-7nb9d\" (UID: \"8d251d9b-c5e7-46e7-b5ab-524b8171d83b\") " pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:35.669982 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669905    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8ppp\" (UniqueName: \"kubernetes.io/projected/8da6593d-d284-438d-96ee-ec0d9ddf1124-kube-api-access-s8ppp\") pod \"node-ca-2kzdk\" (UID: \"8da6593d-d284-438d-96ee-ec0d9ddf1124\") " pod="openshift-image-registry/node-ca-2kzdk"
Oct 31 03:19:35.669982 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.669959    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-run-ovn-kubernetes\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.670115 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670062    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjkmq\" (UniqueName: \"kubernetes.io/projected/7d72ff8d-4480-4266-ab8b-a225f4245836-kube-api-access-jjkmq\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.670115 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670092    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-tmp\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.670168 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670159    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-tmp\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.670198 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670162    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-run\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.670198 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670192    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-var-lib-cni-multus\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.670250 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670214    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ac9dbe77-127f-4a88-9634-82560e6424ce-config-volume\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:35.670250 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670234    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-systemd-units\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.670305 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670254    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74face88-7a89-4b8f-9e8b-adea29761ed8-cni-binary-copy\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.670305 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670291    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-run-k8s-cni-cncf-io\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.670357 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670321    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zzzr\" (UniqueName: \"kubernetes.io/projected/74face88-7a89-4b8f-9e8b-adea29761ed8-kube-api-access-5zzzr\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.670357 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670336    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d72ff8d-4480-4266-ab8b-a225f4245836-env-overrides\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.670410 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670353    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-system-cni-dir\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:19:35.670410 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670380    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-sys\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.670410 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670406    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-multus-cni-dir\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.670521 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670426    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:35.670521 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670447    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:19:35.670521 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670477    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8da6593d-d284-438d-96ee-ec0d9ddf1124-host\") pod \"node-ca-2kzdk\" (UID: \"8da6593d-d284-438d-96ee-ec0d9ddf1124\") " pod="openshift-image-registry/node-ca-2kzdk"
Oct 31 03:19:35.670521 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670514    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-run-netns\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.670629 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670550    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-cnibin\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:19:35.670629 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670577    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-socket-dir\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:19:35.670629 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670605    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-tuned\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.670629 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670624    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4wv2\" (UniqueName: \"kubernetes.io/projected/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-kube-api-access-n4wv2\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.670727 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670665    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-modprobe-d\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.670727 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670697    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-tuned\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.670727 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670709    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-lib-modules\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.670805 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670739    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d72ff8d-4480-4266-ab8b-a225f4245836-ovn-node-metrics-cert\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.670805 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670775    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8da6593d-d284-438d-96ee-ec0d9ddf1124-serviceca\") pod \"node-ca-2kzdk\" (UID: \"8da6593d-d284-438d-96ee-ec0d9ddf1124\") " pod="openshift-image-registry/node-ca-2kzdk"
Oct 31 03:19:35.670805 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670794    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-sysctl-d\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.670914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670809    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-multus-socket-dir-parent\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.670914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670825    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d72ff8d-4480-4266-ab8b-a225f4245836-ovnkube-script-lib\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.670914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670848    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be4c5472-30cf-447a-85fe-b90f5f8b022a-crio-socket\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:35.670914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670869    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be4c5472-30cf-447a-85fe-b90f5f8b022a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:35.670914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670908    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hjs5\" (UniqueName: \"kubernetes.io/projected/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-kube-api-access-2hjs5\") pod \"network-metrics-daemon-bx98q\" (UID: \"45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9\") " pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:19:35.671043 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670926    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:19:35.671043 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.670973    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-kubelet-config\") pod \"global-pull-secret-syncer-7nb9d\" (UID: \"8d251d9b-c5e7-46e7-b5ab-524b8171d83b\") " pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:35.701315 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.701292    4447 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:19:35.701411 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:19:35.701353    4447 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:19:35.701965 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.701951    4447 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:19:35.702005 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:19:35.701992    4447 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.us-west-1.compute.internal"
Oct 31 03:19:35.709984 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.709963    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v5ddh\""
Oct 31 03:19:35.710076 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.709963    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-vxflf\""
Oct 31 03:19:35.710118 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.710101    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hkxjs\""
Oct 31 03:19:35.710167 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.710101    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 03:19:35.713049 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.713033    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 03:19:35.713049 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.713038    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 03:19:35.713193 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.713053    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 03:19:35.713193 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.713095    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 03:19:35.713193 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.713171    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 03:19:35.713193 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.713173    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 03:19:35.713193 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.713172    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 03:19:35.713409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.713203    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 03:19:35.713409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.713222    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 03:19:35.713409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.713272    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 03:19:35.713409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.713294    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 03:19:35.713409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.713323    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 03:19:35.713409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.713333    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 03:19:35.713409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.713337    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 03:19:35.713409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.713376    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 03:19:35.713409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.713401    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 03:19:35.713749 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.713542    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 03:19:35.718750 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.718733    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/74face88-7a89-4b8f-9e8b-adea29761ed8-multus-daemon-config\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.718918 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.718904    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:19:35.719729 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.719714    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-cni-binary-copy\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:19:35.719819 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.719735    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ff0da67d-c75e-4487-ba6a-b85e04a496d9-konnectivity-ca\") pod \"konnectivity-agent-kqgxt\" (UID: \"ff0da67d-c75e-4487-ba6a-b85e04a496d9\") " pod="kube-system/konnectivity-agent-kqgxt"
Oct 31 03:19:35.720122 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.719917    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d72ff8d-4480-4266-ab8b-a225f4245836-ovnkube-config\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.720814 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.720798    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d72ff8d-4480-4266-ab8b-a225f4245836-env-overrides\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.720908 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.720812    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:19:35.720908 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.720850    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74face88-7a89-4b8f-9e8b-adea29761ed8-cni-binary-copy\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.721062 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.721049    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d72ff8d-4480-4266-ab8b-a225f4245836-ovnkube-script-lib\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.754487 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.754453    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hjs5\" (UniqueName: \"kubernetes.io/projected/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-kube-api-access-2hjs5\") pod \"network-metrics-daemon-bx98q\" (UID: \"45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9\") " pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:19:35.754487 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.754475    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zzzr\" (UniqueName: \"kubernetes.io/projected/74face88-7a89-4b8f-9e8b-adea29761ed8-kube-api-access-5zzzr\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.754670 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.754656    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjkmq\" (UniqueName: \"kubernetes.io/projected/7d72ff8d-4480-4266-ab8b-a225f4245836-kube-api-access-jjkmq\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.754745 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.754729    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5jxx\" (UniqueName: \"kubernetes.io/projected/bf1633f9-bdb5-4a26-bd80-a83a4b5587e8-kube-api-access-p5jxx\") pod \"network-check-target-rtp6w\" (UID: \"bf1633f9-bdb5-4a26-bd80-a83a4b5587e8\") " pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:19:35.755359 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.755339    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2jf8\" (UniqueName: \"kubernetes.io/projected/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-kube-api-access-b2jf8\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:19:35.755438 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.755340    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp4mc\" (UniqueName: \"kubernetes.io/projected/ebb8e08d-8243-492f-aef5-d3b6590d9f47-kube-api-access-cp4mc\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:19:35.765034 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.765017    4447 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 03:19:35.771849 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.771829    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-run-k8s-cni-cncf-io\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.771940 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.771856    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-system-cni-dir\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:19:35.771940 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.771876    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-sys\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.771940 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.771934    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-system-cni-dir\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:19:35.772108 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.771958    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-multus-cni-dir\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.772108 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.771976    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:35.772108 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.771949    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-run-k8s-cni-cncf-io\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.772108 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.771994    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8da6593d-d284-438d-96ee-ec0d9ddf1124-host\") pod \"node-ca-2kzdk\" (UID: \"8da6593d-d284-438d-96ee-ec0d9ddf1124\") " pod="openshift-image-registry/node-ca-2kzdk"
Oct 31 03:19:35.772108 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772033    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-sys\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.772108 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772040    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-multus-cni-dir\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.772108 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772061    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-run-netns\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.772108 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772103    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-cnibin\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:19:35.772318 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772046    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8da6593d-d284-438d-96ee-ec0d9ddf1124-host\") pod \"node-ca-2kzdk\" (UID: \"8da6593d-d284-438d-96ee-ec0d9ddf1124\") " pod="openshift-image-registry/node-ca-2kzdk"
Oct 31 03:19:35.772318 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772125    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-socket-dir\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:19:35.772318 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772135    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-run-netns\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.772318 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772158    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-cnibin\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:19:35.772318 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772162    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a0f3be01-6bec-4a39-980a-e6342fcb3624-signing-cabundle\") pod \"service-ca-5fffc97bdf-tqfr6\" (UID: \"a0f3be01-6bec-4a39-980a-e6342fcb3624\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tqfr6"
Oct 31 03:19:35.772318 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772170    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-socket-dir\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:19:35.772318 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772223    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-modprobe-d\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.772318 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772249    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-lib-modules\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.772318 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772302    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-sysctl-d\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.772541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772352    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-multus-socket-dir-parent\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.772541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772361    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-modprobe-d\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.772541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772375    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be4c5472-30cf-447a-85fe-b90f5f8b022a-crio-socket\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:35.772541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772401    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:19:35.772541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772414    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-multus-socket-dir-parent\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.772541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772421    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-sysctl-d\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.772541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772423    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-kubelet-config\") pod \"global-pull-secret-syncer-7nb9d\" (UID: \"8d251d9b-c5e7-46e7-b5ab-524b8171d83b\") " pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:35.772541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772401    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-lib-modules\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.772541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772453    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:19:35.772541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772461    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be4c5472-30cf-447a-85fe-b90f5f8b022a-crio-socket\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:35.772541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772449    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-sysconfig\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.772541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772492    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-kubernetes\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.772541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772487    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-kubelet-config\") pod \"global-pull-secret-syncer-7nb9d\" (UID: \"8d251d9b-c5e7-46e7-b5ab-524b8171d83b\") " pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:35.772541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772500    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-sysconfig\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.772541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772508    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-system-cni-dir\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.772541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772540    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-system-cni-dir\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.772541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772541    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-kubernetes\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.772541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772546    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-sys-fs\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772589    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-sys-fs\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772607    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-var-lib-kubelet\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772623    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-cnibin\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772634    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-var-lib-kubelet\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772643    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-var-lib-kubelet\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772664    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-cnibin\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772669    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-slash\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772692    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-slash\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772696    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-var-lib-kubelet\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772717    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-run-netns\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772734    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-systemd\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772751    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-run-multus-certs\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772776    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce-hosts-file\") pod \"node-resolver-j86qh\" (UID: \"65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce\") " pod="openshift-dns/node-resolver-j86qh"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772776    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-run-netns\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772792    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-cni-netd\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772792    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-systemd\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772815    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772823    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-run-multus-certs\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772851    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce-hosts-file\") pod \"node-resolver-j86qh\" (UID: \"65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce\") " pod="openshift-dns/node-resolver-j86qh"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772853    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-cni-netd\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772862    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772854    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772898    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcb6233e-08f7-4a15-b273-b705bf9c2ac7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lkj57\" (UID: \"bcb6233e-08f7-4a15-b273-b705bf9c2ac7\") " pod="openshift-multus/multus-additional-cni-plugins-lkj57"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772905    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs\") pod \"network-metrics-daemon-bx98q\" (UID: \"45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9\") " pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772932    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-registration-dir\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772959    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-etc-selinux\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772974    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-multus-conf-dir\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.772993    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-kubelet\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773008    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-registration-dir\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:19:35.773029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773014    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-run-systemd\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773046    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-etc-selinux\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773065    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-kubelet\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773078    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-multus-conf-dir\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773086    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-var-lib-openvswitch\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773106    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-run-systemd\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773115    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-etc-kubernetes\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773133    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-run-ovn\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773155    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-node-log\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773176    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-var-lib-openvswitch\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773181    4447 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773178    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-cni-bin\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773185    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-etc-kubernetes\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773210    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-cni-bin\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773213    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-node-log\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773194    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-run-ovn\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773279    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773323    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-hostroot\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773371    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-dbus\") pod \"global-pull-secret-syncer-7nb9d\" (UID: \"8d251d9b-c5e7-46e7-b5ab-524b8171d83b\") " pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773397    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-host\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773405    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-dbus\") pod \"global-pull-secret-syncer-7nb9d\" (UID: \"8d251d9b-c5e7-46e7-b5ab-524b8171d83b\") " pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773372    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-hostroot\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773420    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2aa25e13-aaa4-4795-a221-2056b76e8c8f-host-slash\") pod \"iptables-alerter-9x2kw\" (UID: \"2aa25e13-aaa4-4795-a221-2056b76e8c8f\") " pod="openshift-network-operator/iptables-alerter-9x2kw"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773455    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-host\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773467    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2aa25e13-aaa4-4795-a221-2056b76e8c8f-host-slash\") pod \"iptables-alerter-9x2kw\" (UID: \"2aa25e13-aaa4-4795-a221-2056b76e8c8f\") " pod="openshift-network-operator/iptables-alerter-9x2kw"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773483    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-sysctl-conf\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773531    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a0f3be01-6bec-4a39-980a-e6342fcb3624-signing-key\") pod \"service-ca-5fffc97bdf-tqfr6\" (UID: \"a0f3be01-6bec-4a39-980a-e6342fcb3624\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tqfr6"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773558    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-var-lib-cni-bin\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773578    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-run-openvswitch\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773595    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-etc-sysctl-conf\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773600    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-log-socket\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773614    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-var-lib-cni-bin\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773624    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-device-dir\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773635    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-run-openvswitch\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773658    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-etc-openvswitch\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773664    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-log-socket\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773706    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ebb8e08d-8243-492f-aef5-d3b6590d9f47-device-dir\") pod \"aws-ebs-csi-driver-node-x2rnp\" (UID: \"ebb8e08d-8243-492f-aef5-d3b6590d9f47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2rnp"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773707    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-etc-openvswitch\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773706    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-run-ovn-kubernetes\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773738    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-host-run-ovn-kubernetes\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773764    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjh4p\" (UniqueName: \"kubernetes.io/projected/a0f3be01-6bec-4a39-980a-e6342fcb3624-kube-api-access-cjh4p\") pod \"service-ca-5fffc97bdf-tqfr6\" (UID: \"a0f3be01-6bec-4a39-980a-e6342fcb3624\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tqfr6"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773787    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-run\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773802    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-var-lib-cni-multus\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773818    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-systemd-units\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773852    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d72ff8d-4480-4266-ab8b-a225f4245836-systemd-units\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773863    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-run\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.774165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.773922    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/74face88-7a89-4b8f-9e8b-adea29761ed8-host-var-lib-cni-multus\") pod \"multus-vxkwg\" (UID: \"74face88-7a89-4b8f-9e8b-adea29761ed8\") " pod="openshift-multus/multus-vxkwg"
Oct 31 03:19:35.775355 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.775300    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9-metrics-certs\") pod \"network-metrics-daemon-bx98q\" (UID: \"45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9\") " pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:19:35.809674 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.809644    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 03:19:35.809835 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.809780    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 03:19:35.809952 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.809850    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 03:19:35.810092 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.810064    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 03:19:35.810303 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.810285    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xz7gs\""
Oct 31 03:19:35.810397 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.810371    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-btzr7\""
Oct 31 03:19:35.810439 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.810422    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dbgqm\""
Oct 31 03:19:35.810652 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.810602    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.810813    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.810869    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.810812    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vknbj\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.810920    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ltpjd\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.810928    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xmnbh\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.810984    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811012    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811024    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-n6jvn\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811032    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811054    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811069    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811075    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811088    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-9x6rv\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811123    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811094    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811142    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811121    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811158    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811169    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811174    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811182    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811195    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811215    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sv7k2\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811230    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4kwqf\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811264    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811267    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811286    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4wd7m\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811289    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d72ff8d-4480-4266-ab8b-a225f4245836-ovn-node-metrics-cert\") pod \"ovnkube-node-rkkr4\" (UID: \"7d72ff8d-4480-4266-ab8b-a225f4245836\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811301    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811329    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811335    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-b4ln2\""
Oct 31 03:19:35.811378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811399    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-67ksm\""
Oct 31 03:19:35.812287 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.811660    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8da6593d-d284-438d-96ee-ec0d9ddf1124-serviceca\") pod \"node-ca-2kzdk\" (UID: \"8da6593d-d284-438d-96ee-ec0d9ddf1124\") " pod="openshift-image-registry/node-ca-2kzdk"
Oct 31 03:19:35.812525 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:19:35.812512    4447 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:19:35.812553 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.812531    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a0f3be01-6bec-4a39-980a-e6342fcb3624-signing-cabundle\") pod \"service-ca-5fffc97bdf-tqfr6\" (UID: \"a0f3be01-6bec-4a39-980a-e6342fcb3624\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tqfr6"
Oct 31 03:19:35.812644 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:19:35.812634    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls podName:ac9dbe77-127f-4a88-9634-82560e6424ce nodeName:}" failed. No retries permitted until 2026-10-31 03:19:36.312619489 +0000 UTC m=+1.823601865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls") pod "dns-default-zxbcc" (UID: "ac9dbe77-127f-4a88-9634-82560e6424ce") : secret "dns-default-metrics-tls" not found
Oct 31 03:19:35.813666 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:19:35.813654    4447 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:19:35.813704 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:19:35.813698    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls podName:be4c5472-30cf-447a-85fe-b90f5f8b022a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:36.313679496 +0000 UTC m=+1.824661867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qbl5h" (UID: "be4c5472-30cf-447a-85fe-b90f5f8b022a") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:19:35.813737 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.813728    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a0f3be01-6bec-4a39-980a-e6342fcb3624-signing-key\") pod \"service-ca-5fffc97bdf-tqfr6\" (UID: \"a0f3be01-6bec-4a39-980a-e6342fcb3624\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tqfr6"
Oct 31 03:19:35.815379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.815284    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxphc\" (UniqueName: \"kubernetes.io/projected/ac9dbe77-127f-4a88-9634-82560e6424ce-kube-api-access-kxphc\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:35.815379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.815282    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8ppp\" (UniqueName: \"kubernetes.io/projected/8da6593d-d284-438d-96ee-ec0d9ddf1124-kube-api-access-s8ppp\") pod \"node-ca-2kzdk\" (UID: \"8da6593d-d284-438d-96ee-ec0d9ddf1124\") " pod="openshift-image-registry/node-ca-2kzdk"
Oct 31 03:19:35.815379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.815287    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ln55\" (UniqueName: \"kubernetes.io/projected/be4c5472-30cf-447a-85fe-b90f5f8b022a-kube-api-access-5ln55\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:35.815568 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.815416    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j92zc\" (UniqueName: \"kubernetes.io/projected/65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce-kube-api-access-j92zc\") pod \"node-resolver-j86qh\" (UID: \"65c7fc32-ca2a-43f3-a2e6-fe23ee3480ce\") " pod="openshift-dns/node-resolver-j86qh"
Oct 31 03:19:35.815568 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.815419    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4wv2\" (UniqueName: \"kubernetes.io/projected/95bd2ef0-32a9-4150-abee-d2a9e03cc62c-kube-api-access-n4wv2\") pod \"tuned-l494z\" (UID: \"95bd2ef0-32a9-4150-abee-d2a9e03cc62c\") " pod="openshift-cluster-node-tuning-operator/tuned-l494z"
Oct 31 03:19:35.815568 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.815511    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tpl7\" (UniqueName: \"kubernetes.io/projected/2aa25e13-aaa4-4795-a221-2056b76e8c8f-kube-api-access-6tpl7\") pod \"iptables-alerter-9x2kw\" (UID: \"2aa25e13-aaa4-4795-a221-2056b76e8c8f\") " pod="openshift-network-operator/iptables-alerter-9x2kw"
Oct 31 03:19:35.815568 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.815547    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz5wr\" (UniqueName: \"kubernetes.io/projected/a7360ff6-d326-485b-bd3d-fd2f0f9099b5-kube-api-access-kz5wr\") pod \"ingress-canary-r9prp\" (UID: \"a7360ff6-d326-485b-bd3d-fd2f0f9099b5\") " pod="openshift-ingress-canary/ingress-canary-r9prp"
Oct 31 03:19:35.819308 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.819288    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7360ff6-d326-485b-bd3d-fd2f0f9099b5-cert\") pod \"ingress-canary-r9prp\" (UID: \"a7360ff6-d326-485b-bd3d-fd2f0f9099b5\") " pod="openshift-ingress-canary/ingress-canary-r9prp"
Oct 31 03:19:35.819392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.819335    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ff0da67d-c75e-4487-ba6a-b85e04a496d9-agent-certs\") pod \"konnectivity-agent-kqgxt\" (UID: \"ff0da67d-c75e-4487-ba6a-b85e04a496d9\") " pod="kube-system/konnectivity-agent-kqgxt"
Oct 31 03:19:35.819449 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.819401    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2aa25e13-aaa4-4795-a221-2056b76e8c8f-iptables-alerter-script\") pod \"iptables-alerter-9x2kw\" (UID: \"2aa25e13-aaa4-4795-a221-2056b76e8c8f\") " pod="openshift-network-operator/iptables-alerter-9x2kw"
Oct 31 03:19:35.820690 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.820673    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8d251d9b-c5e7-46e7-b5ab-524b8171d83b-original-pull-secret\") pod \"global-pull-secret-syncer-7nb9d\" (UID: \"8d251d9b-c5e7-46e7-b5ab-524b8171d83b\") " pod="kube-system/global-pull-secret-syncer-7nb9d"
Oct 31 03:19:35.820745 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.820713    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ac9dbe77-127f-4a88-9634-82560e6424ce-config-volume\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:35.822030 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.822016    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be4c5472-30cf-447a-85fe-b90f5f8b022a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:35.838767 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.838741    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjh4p\" (UniqueName: \"kubernetes.io/projected/a0f3be01-6bec-4a39-980a-e6342fcb3624-kube-api-access-cjh4p\") pod \"service-ca-5fffc97bdf-tqfr6\" (UID: \"a0f3be01-6bec-4a39-980a-e6342fcb3624\") " pod="openshift-service-ca/service-ca-5fffc97bdf-tqfr6"
Oct 31 03:19:35.957232 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:35.957157    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx98q"
Oct 31 03:19:36.366527 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:36.366451    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bx98q"]
Oct 31 03:19:36.369624 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:36.369596    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d94fb5_7dbb_4f90_b8a8_8ab0f305a5b9.slice/crio-29927866db5e2ac7cd7a32f8cf03cf27308ae6ce5032827d93c219f34e52a9c9 WatchSource:0}: Error finding container 29927866db5e2ac7cd7a32f8cf03cf27308ae6ce5032827d93c219f34e52a9c9: Status 404 returned error can't find the container with id 29927866db5e2ac7cd7a32f8cf03cf27308ae6ce5032827d93c219f34e52a9c9
Oct 31 03:19:36.371640 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:36.371624    4447 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:19:36.378704 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:36.378679    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:36.378812 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:36.378748    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:36.378868 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:19:36.378844    4447 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:19:36.378868 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:19:36.378851    4447 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:19:36.378977 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:19:36.378925    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls podName:ac9dbe77-127f-4a88-9634-82560e6424ce nodeName:}" failed. No retries permitted until 2026-10-31 03:19:37.37890309 +0000 UTC m=+2.889885459 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls") pod "dns-default-zxbcc" (UID: "ac9dbe77-127f-4a88-9634-82560e6424ce") : secret "dns-default-metrics-tls" not found
Oct 31 03:19:36.378977 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:19:36.378952    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls podName:be4c5472-30cf-447a-85fe-b90f5f8b022a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:37.378933516 +0000 UTC m=+2.889915888 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qbl5h" (UID: "be4c5472-30cf-447a-85fe-b90f5f8b022a") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:19:36.643365 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:36.643280    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bx98q" event={"ID":"45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9","Type":"ContainerStarted","Data":"29927866db5e2ac7cd7a32f8cf03cf27308ae6ce5032827d93c219f34e52a9c9"}
Oct 31 03:19:37.386542 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:37.386502    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:37.386739 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:37.386573    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:37.386739 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:19:37.386667    4447 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:19:37.386739 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:19:37.386667    4447 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:19:37.386739 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:19:37.386734    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls podName:be4c5472-30cf-447a-85fe-b90f5f8b022a nodeName:}" failed. No retries permitted until 2026-10-31 03:19:39.386718939 +0000 UTC m=+4.897701297 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qbl5h" (UID: "be4c5472-30cf-447a-85fe-b90f5f8b022a") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:19:37.386927 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:19:37.386748    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls podName:ac9dbe77-127f-4a88-9634-82560e6424ce nodeName:}" failed. No retries permitted until 2026-10-31 03:19:39.386741011 +0000 UTC m=+4.897723366 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls") pod "dns-default-zxbcc" (UID: "ac9dbe77-127f-4a88-9634-82560e6424ce") : secret "dns-default-metrics-tls" not found
Oct 31 03:19:37.461036 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:37.461002    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:19:38.653595 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:38.653562    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bx98q" event={"ID":"45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9","Type":"ContainerStarted","Data":"505a766583c468cb3f86a101a4b4bc07b242eb6369ee61f3bea8ea56a32e264f"}
Oct 31 03:19:38.654051 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:38.653613    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bx98q" event={"ID":"45d94fb5-7dbb-4f90-b8a8-8ab0f305a5b9","Type":"ContainerStarted","Data":"c4d5f6cc2dbb326e78a8bba4fc2650ace647ae112e74ffdb4b011278572e633e"}
Oct 31 03:19:39.404988 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:39.404947    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:39.405308 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:39.405009    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:39.407454 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:39.407426    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac9dbe77-127f-4a88-9634-82560e6424ce-metrics-tls\") pod \"dns-default-zxbcc\" (UID: \"ac9dbe77-127f-4a88-9634-82560e6424ce\") " pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:39.407579 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:39.407424    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be4c5472-30cf-447a-85fe-b90f5f8b022a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbl5h\" (UID: \"be4c5472-30cf-447a-85fe-b90f5f8b022a\") " pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:39.564516 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:39.564487    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qbl5h"
Oct 31 03:19:39.564724 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:39.564441    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:39.992976 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:39.992947    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qbl5h"]
Oct 31 03:19:39.995582 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:39.995561    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zxbcc"]
Oct 31 03:19:39.996117 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:39.996096    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4c5472_30cf_447a_85fe_b90f5f8b022a.slice/crio-391db56df6a7f26d0e11209db8010f1a5b7fc6a1fe159b0924a1365f0c616562 WatchSource:0}: Error finding container 391db56df6a7f26d0e11209db8010f1a5b7fc6a1fe159b0924a1365f0c616562: Status 404 returned error can't find the container with id 391db56df6a7f26d0e11209db8010f1a5b7fc6a1fe159b0924a1365f0c616562
Oct 31 03:19:40.034451 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:40.034425    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac9dbe77_127f_4a88_9634_82560e6424ce.slice/crio-96a40d0a321ae62f12a5544d790d2174ace11d3dcc6dc0bb915801f51434200c WatchSource:0}: Error finding container 96a40d0a321ae62f12a5544d790d2174ace11d3dcc6dc0bb915801f51434200c: Status 404 returned error can't find the container with id 96a40d0a321ae62f12a5544d790d2174ace11d3dcc6dc0bb915801f51434200c
Oct 31 03:19:40.662411 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:40.662374    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zxbcc" event={"ID":"ac9dbe77-127f-4a88-9634-82560e6424ce","Type":"ContainerStarted","Data":"96a40d0a321ae62f12a5544d790d2174ace11d3dcc6dc0bb915801f51434200c"}
Oct 31 03:19:40.664909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:40.664859    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qbl5h" event={"ID":"be4c5472-30cf-447a-85fe-b90f5f8b022a","Type":"ContainerStarted","Data":"a76b3a0681448ce941cffd8200116c3ae613c6575d79a1b1f1ccfbf4c66b73f2"}
Oct 31 03:19:40.665070 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:40.664979    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qbl5h" event={"ID":"be4c5472-30cf-447a-85fe-b90f5f8b022a","Type":"ContainerStarted","Data":"391db56df6a7f26d0e11209db8010f1a5b7fc6a1fe159b0924a1365f0c616562"}
Oct 31 03:19:41.670301 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:41.670259    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qbl5h" event={"ID":"be4c5472-30cf-447a-85fe-b90f5f8b022a","Type":"ContainerStarted","Data":"ebced328caed775dff45a89ff3a93862a76740b5d48a052bfab7d4ce7eb30754"}
Oct 31 03:19:42.187001 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:42.186975    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:42.676625 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:42.676585    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zxbcc" event={"ID":"ac9dbe77-127f-4a88-9634-82560e6424ce","Type":"ContainerStarted","Data":"89e41cca4381c2ca67fecaacea27400217eb90d7eb5a3bc32e0c90dd87ade56f"}
Oct 31 03:19:42.677077 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:42.676648    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zxbcc" event={"ID":"ac9dbe77-127f-4a88-9634-82560e6424ce","Type":"ContainerStarted","Data":"7eb23f84b3fa9b182919fada0adbf16d589b15aa53e87c4a9be7080434878dae"}
Oct 31 03:19:42.677077 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:42.676705    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:43.682034 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:43.681943    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qbl5h" event={"ID":"be4c5472-30cf-447a-85fe-b90f5f8b022a","Type":"ContainerStarted","Data":"4e715ed5630a43d8a6c3b0e15d88b9a69ec9b9d9cd83ce0af4fbbc040a90ce3b"}
Oct 31 03:19:45.226120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:45.226087    4447 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-255.us-west-1.compute.internal" event="NodeReady"
Oct 31 03:19:45.670696 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:45.670620    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:45.670856 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:45.670842    4447 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 03:19:49.000022 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:19:48.999977    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:19:49.542021 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:49.541993    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-r4qz8"]
Oct 31 03:19:49.543648 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:49.543633    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r4qz8"
Oct 31 03:19:49.579170 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:49.579135    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09708991-5135-48b2-9615-1b40d5375e63-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-r4qz8\" (UID: \"09708991-5135-48b2-9615-1b40d5375e63\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r4qz8"
Oct 31 03:19:49.579326 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:49.579180    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkc72\" (UniqueName: \"kubernetes.io/projected/09708991-5135-48b2-9615-1b40d5375e63-kube-api-access-xkc72\") pod \"prometheus-operator-9cd89b74c-r4qz8\" (UID: \"09708991-5135-48b2-9615-1b40d5375e63\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r4qz8"
Oct 31 03:19:49.579326 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:49.579249    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09708991-5135-48b2-9615-1b40d5375e63-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-r4qz8\" (UID: \"09708991-5135-48b2-9615-1b40d5375e63\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r4qz8"
Oct 31 03:19:49.579435 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:49.579329    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/09708991-5135-48b2-9615-1b40d5375e63-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-r4qz8\" (UID: \"09708991-5135-48b2-9615-1b40d5375e63\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r4qz8"
Oct 31 03:19:49.596558 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:49.596529    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 31 03:19:49.646640 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:49.646599    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 03:19:49.646640 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:49.646600    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 31 03:19:49.646842 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:49.646611    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 03:19:49.646842 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:49.646675    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 03:19:49.646842 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:49.646703    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7fhsq\""
Oct 31 03:19:49.680195 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:49.680163    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkc72\" (UniqueName: \"kubernetes.io/projected/09708991-5135-48b2-9615-1b40d5375e63-kube-api-access-xkc72\") pod \"prometheus-operator-9cd89b74c-r4qz8\" (UID: \"09708991-5135-48b2-9615-1b40d5375e63\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r4qz8"
Oct 31 03:19:49.680307 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:49.680205    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09708991-5135-48b2-9615-1b40d5375e63-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-r4qz8\" (UID: \"09708991-5135-48b2-9615-1b40d5375e63\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r4qz8"
Oct 31 03:19:49.680307 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:49.680250    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/09708991-5135-48b2-9615-1b40d5375e63-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-r4qz8\" (UID: \"09708991-5135-48b2-9615-1b40d5375e63\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r4qz8"
Oct 31 03:19:49.680307 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:49.680283    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09708991-5135-48b2-9615-1b40d5375e63-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-r4qz8\" (UID: \"09708991-5135-48b2-9615-1b40d5375e63\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r4qz8"
Oct 31 03:19:49.680955 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:49.680936    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09708991-5135-48b2-9615-1b40d5375e63-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-r4qz8\" (UID: \"09708991-5135-48b2-9615-1b40d5375e63\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r4qz8"
Oct 31 03:19:49.682631 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:49.682604    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09708991-5135-48b2-9615-1b40d5375e63-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-r4qz8\" (UID: \"09708991-5135-48b2-9615-1b40d5375e63\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r4qz8"
Oct 31 03:19:49.682826 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:49.682804    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/09708991-5135-48b2-9615-1b40d5375e63-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-r4qz8\" (UID: \"09708991-5135-48b2-9615-1b40d5375e63\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r4qz8"
Oct 31 03:19:49.696953 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:49.696932    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-r4qz8"]
Oct 31 03:19:49.741606 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:49.741576    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkc72\" (UniqueName: \"kubernetes.io/projected/09708991-5135-48b2-9615-1b40d5375e63-kube-api-access-xkc72\") pod \"prometheus-operator-9cd89b74c-r4qz8\" (UID: \"09708991-5135-48b2-9615-1b40d5375e63\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r4qz8"
Oct 31 03:19:49.854425 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:49.854337    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r4qz8"
Oct 31 03:19:50.220267 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:50.220240    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-r4qz8"]
Oct 31 03:19:50.252252 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:50.252223    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09708991_5135_48b2_9615_1b40d5375e63.slice/crio-3e8e203de5c6ed28790b31b673e04598caa98d7298b239c83c7d4eaf8423647f WatchSource:0}: Error finding container 3e8e203de5c6ed28790b31b673e04598caa98d7298b239c83c7d4eaf8423647f: Status 404 returned error can't find the container with id 3e8e203de5c6ed28790b31b673e04598caa98d7298b239c83c7d4eaf8423647f
Oct 31 03:19:50.703156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:50.703070    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r4qz8" event={"ID":"09708991-5135-48b2-9615-1b40d5375e63","Type":"ContainerStarted","Data":"3e8e203de5c6ed28790b31b673e04598caa98d7298b239c83c7d4eaf8423647f"}
Oct 31 03:19:52.684683 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:52.684659    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zxbcc"
Oct 31 03:19:52.711365 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:52.711333    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r4qz8" event={"ID":"09708991-5135-48b2-9615-1b40d5375e63","Type":"ContainerStarted","Data":"900acfd1d741706b3b72c6ba2ab660d16fa9ffa619bf3dd5f72607019df34649"}
Oct 31 03:19:52.711508 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:52.711379    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r4qz8" event={"ID":"09708991-5135-48b2-9615-1b40d5375e63","Type":"ContainerStarted","Data":"274f6b0e28493e7be9a377d3084edac82382db3fcc403562a44c57fffee5dfac"}
Oct 31 03:19:52.920122 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:52.920064    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r4qz8" podStartSLOduration=1.9824391449999998 podStartE2EDuration="3.920049776s" totalImagesPullingTime="1.937610631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:52.919246121 +0000 UTC m=+18.430228499" watchObservedRunningTime="2026-10-31 03:19:52.920049776 +0000 UTC m=+18.431032150"
Oct 31 03:19:55.018342 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:55.018306    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rkkr4"
Oct 31 03:19:56.125403 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.125366    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kqgxt"
Oct 31 03:19:56.348958 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.348921    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb"]
Oct 31 03:19:56.351218 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.351203    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8gkd5"]
Oct 31 03:19:56.351334 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.351317    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb"
Oct 31 03:19:56.353086 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.353064    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.428400 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.428312    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/213aee5f-38c2-41ce-8790-e37ec9437e10-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-h4gsb\" (UID: \"213aee5f-38c2-41ce-8790-e37ec9437e10\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb"
Oct 31 03:19:56.428400 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.428351    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/311bf3de-3730-4e0b-9872-192c9bde06cf-root\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.428400 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.428375    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-866jz\" (UniqueName: \"kubernetes.io/projected/311bf3de-3730-4e0b-9872-192c9bde06cf-kube-api-access-866jz\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.428605 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.428467    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/311bf3de-3730-4e0b-9872-192c9bde06cf-node-exporter-wtmp\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.428605 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.428491    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgpsq\" (UniqueName: \"kubernetes.io/projected/213aee5f-38c2-41ce-8790-e37ec9437e10-kube-api-access-bgpsq\") pod \"kube-state-metrics-5dc8d7d89b-h4gsb\" (UID: \"213aee5f-38c2-41ce-8790-e37ec9437e10\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb"
Oct 31 03:19:56.428605 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.428515    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/213aee5f-38c2-41ce-8790-e37ec9437e10-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-h4gsb\" (UID: \"213aee5f-38c2-41ce-8790-e37ec9437e10\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb"
Oct 31 03:19:56.428605 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.428539    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/311bf3de-3730-4e0b-9872-192c9bde06cf-node-exporter-tls\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.428605 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.428575    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/213aee5f-38c2-41ce-8790-e37ec9437e10-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-h4gsb\" (UID: \"213aee5f-38c2-41ce-8790-e37ec9437e10\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb"
Oct 31 03:19:56.428795 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.428605    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/311bf3de-3730-4e0b-9872-192c9bde06cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.428795 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.428689    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/311bf3de-3730-4e0b-9872-192c9bde06cf-sys\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.428795 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.428710    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/311bf3de-3730-4e0b-9872-192c9bde06cf-metrics-client-ca\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.428795 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.428726    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/213aee5f-38c2-41ce-8790-e37ec9437e10-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-h4gsb\" (UID: \"213aee5f-38c2-41ce-8790-e37ec9437e10\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb"
Oct 31 03:19:56.428795 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.428742    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/311bf3de-3730-4e0b-9872-192c9bde06cf-node-exporter-textfile\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.428795 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.428773    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/311bf3de-3730-4e0b-9872-192c9bde06cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.428992 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.428805    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/213aee5f-38c2-41ce-8790-e37ec9437e10-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-h4gsb\" (UID: \"213aee5f-38c2-41ce-8790-e37ec9437e10\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb"
Oct 31 03:19:56.437033 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.436995    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb"]
Oct 31 03:19:56.454194 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.454169    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 03:19:56.454194 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.454169    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 31 03:19:56.486669 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.486640    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 31 03:19:56.486669 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.486642    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 03:19:56.486669 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.486668    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m85g4\""
Oct 31 03:19:56.504134 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.504108    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 03:19:56.504134 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.504108    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-n9dn6\""
Oct 31 03:19:56.504290 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.504157    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 31 03:19:56.529626 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.529593    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/311bf3de-3730-4e0b-9872-192c9bde06cf-sys\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.529756 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.529628    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/311bf3de-3730-4e0b-9872-192c9bde06cf-metrics-client-ca\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.529756 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.529648    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/213aee5f-38c2-41ce-8790-e37ec9437e10-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-h4gsb\" (UID: \"213aee5f-38c2-41ce-8790-e37ec9437e10\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb"
Oct 31 03:19:56.529756 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.529665    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/311bf3de-3730-4e0b-9872-192c9bde06cf-node-exporter-textfile\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.529756 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.529715    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/311bf3de-3730-4e0b-9872-192c9bde06cf-sys\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.530181 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.530082    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/311bf3de-3730-4e0b-9872-192c9bde06cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.530181 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.530139    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/213aee5f-38c2-41ce-8790-e37ec9437e10-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-h4gsb\" (UID: \"213aee5f-38c2-41ce-8790-e37ec9437e10\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb"
Oct 31 03:19:56.530357 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.530188    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/213aee5f-38c2-41ce-8790-e37ec9437e10-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-h4gsb\" (UID: \"213aee5f-38c2-41ce-8790-e37ec9437e10\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb"
Oct 31 03:19:56.530357 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.530232    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/311bf3de-3730-4e0b-9872-192c9bde06cf-root\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.530357 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.530280    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-866jz\" (UniqueName: \"kubernetes.io/projected/311bf3de-3730-4e0b-9872-192c9bde06cf-kube-api-access-866jz\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.530357 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.530319    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/311bf3de-3730-4e0b-9872-192c9bde06cf-node-exporter-wtmp\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.530357 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.530344    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgpsq\" (UniqueName: \"kubernetes.io/projected/213aee5f-38c2-41ce-8790-e37ec9437e10-kube-api-access-bgpsq\") pod \"kube-state-metrics-5dc8d7d89b-h4gsb\" (UID: \"213aee5f-38c2-41ce-8790-e37ec9437e10\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb"
Oct 31 03:19:56.530597 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.530389    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/213aee5f-38c2-41ce-8790-e37ec9437e10-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-h4gsb\" (UID: \"213aee5f-38c2-41ce-8790-e37ec9437e10\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb"
Oct 31 03:19:56.530597 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.530431    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/311bf3de-3730-4e0b-9872-192c9bde06cf-node-exporter-tls\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.530597 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.530475    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/311bf3de-3730-4e0b-9872-192c9bde06cf-root\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.530747 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.530607    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/311bf3de-3730-4e0b-9872-192c9bde06cf-node-exporter-wtmp\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.530747 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.530641    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/311bf3de-3730-4e0b-9872-192c9bde06cf-metrics-client-ca\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.530948 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.530925    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/311bf3de-3730-4e0b-9872-192c9bde06cf-node-exporter-textfile\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.531402 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.531030    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/213aee5f-38c2-41ce-8790-e37ec9437e10-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-h4gsb\" (UID: \"213aee5f-38c2-41ce-8790-e37ec9437e10\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb"
Oct 31 03:19:56.535399 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.531313    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/213aee5f-38c2-41ce-8790-e37ec9437e10-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-h4gsb\" (UID: \"213aee5f-38c2-41ce-8790-e37ec9437e10\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb"
Oct 31 03:19:56.535498 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.531339    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/213aee5f-38c2-41ce-8790-e37ec9437e10-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-h4gsb\" (UID: \"213aee5f-38c2-41ce-8790-e37ec9437e10\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb"
Oct 31 03:19:56.535552 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.531467    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/213aee5f-38c2-41ce-8790-e37ec9437e10-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-h4gsb\" (UID: \"213aee5f-38c2-41ce-8790-e37ec9437e10\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb"
Oct 31 03:19:56.535552 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.533543    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/311bf3de-3730-4e0b-9872-192c9bde06cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.535657 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.533561    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/311bf3de-3730-4e0b-9872-192c9bde06cf-node-exporter-tls\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.535657 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.533674    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/213aee5f-38c2-41ce-8790-e37ec9437e10-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-h4gsb\" (UID: \"213aee5f-38c2-41ce-8790-e37ec9437e10\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb"
Oct 31 03:19:56.535940 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.535532    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/311bf3de-3730-4e0b-9872-192c9bde06cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.536256 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.536238    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/213aee5f-38c2-41ce-8790-e37ec9437e10-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-h4gsb\" (UID: \"213aee5f-38c2-41ce-8790-e37ec9437e10\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb"
Oct 31 03:19:56.536385 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.536364    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/311bf3de-3730-4e0b-9872-192c9bde06cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.596254 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.596225    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-866jz\" (UniqueName: \"kubernetes.io/projected/311bf3de-3730-4e0b-9872-192c9bde06cf-kube-api-access-866jz\") pod \"node-exporter-8gkd5\" (UID: \"311bf3de-3730-4e0b-9872-192c9bde06cf\") " pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.598981 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.598949    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgpsq\" (UniqueName: \"kubernetes.io/projected/213aee5f-38c2-41ce-8790-e37ec9437e10-kube-api-access-bgpsq\") pod \"kube-state-metrics-5dc8d7d89b-h4gsb\" (UID: \"213aee5f-38c2-41ce-8790-e37ec9437e10\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb"
Oct 31 03:19:56.664090 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.664055    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb"
Oct 31 03:19:56.668379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.668357    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8gkd5"
Oct 31 03:19:56.679509 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:56.679445    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311bf3de_3730_4e0b_9872_192c9bde06cf.slice/crio-891c8711740794ed846e05521059a85d8a18432edf485fe2b11417ef06c9f67c WatchSource:0}: Error finding container 891c8711740794ed846e05521059a85d8a18432edf485fe2b11417ef06c9f67c: Status 404 returned error can't find the container with id 891c8711740794ed846e05521059a85d8a18432edf485fe2b11417ef06c9f67c
Oct 31 03:19:56.725794 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:56.725765    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8gkd5" event={"ID":"311bf3de-3730-4e0b-9872-192c9bde06cf","Type":"ContainerStarted","Data":"891c8711740794ed846e05521059a85d8a18432edf485fe2b11417ef06c9f67c"}
Oct 31 03:19:57.081125 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:57.081004    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb"]
Oct 31 03:19:57.083450 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:57.083419    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213aee5f_38c2_41ce_8790_e37ec9437e10.slice/crio-f174bc77bab752954738a3b4dd5242b572233ae4cee3517e428071d28a7cdcae WatchSource:0}: Error finding container f174bc77bab752954738a3b4dd5242b572233ae4cee3517e428071d28a7cdcae: Status 404 returned error can't find the container with id f174bc77bab752954738a3b4dd5242b572233ae4cee3517e428071d28a7cdcae
Oct 31 03:19:57.730482 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:57.730445    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb" event={"ID":"213aee5f-38c2-41ce-8790-e37ec9437e10","Type":"ContainerStarted","Data":"f174bc77bab752954738a3b4dd5242b572233ae4cee3517e428071d28a7cdcae"}
Oct 31 03:19:58.599320 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.599285    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 03:19:58.601822 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.601797    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.654743 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.654711    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 03:19:58.654743 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.654736    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 03:19:58.654743 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.654712    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 03:19:58.655066 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.654778    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 03:19:58.655066 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.654796    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 03:19:58.704156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.704123    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 03:19:58.704345 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.704169    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5bnqw\""
Oct 31 03:19:58.704345 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.704184    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 03:19:58.705468 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.705441    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 03:19:58.736408 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.736368    4447 generic.go:356] "Generic (PLEG): container finished" podID="311bf3de-3730-4e0b-9872-192c9bde06cf" containerID="4f30fcf462e603e67b59ab6ae09fe6b54785d9292facd958f4121e2aac22094e" exitCode=0
Oct 31 03:19:58.736862 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.736413    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8gkd5" event={"ID":"311bf3de-3730-4e0b-9872-192c9bde06cf","Type":"ContainerDied","Data":"4f30fcf462e603e67b59ab6ae09fe6b54785d9292facd958f4121e2aac22094e"}
Oct 31 03:19:58.753208 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.753172    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-web-config\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.753310 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.753219    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.753381 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.753359    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b20027a-3439-4f84-8407-a9732ee5d748-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.753453 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.753436    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6b20027a-3439-4f84-8407-a9732ee5d748-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.753509 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.753471    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.753509 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.753497    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.753610 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.753588    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-config-volume\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.753643 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.753628    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b20027a-3439-4f84-8407-a9732ee5d748-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.753677 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.753661    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.753718 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.753689    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b20027a-3439-4f84-8407-a9732ee5d748-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.753751 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.753736    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b20027a-3439-4f84-8407-a9732ee5d748-config-out\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.753799 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.753766    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txzbc\" (UniqueName: \"kubernetes.io/projected/6b20027a-3439-4f84-8407-a9732ee5d748-kube-api-access-txzbc\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.753898 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.753871    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.756194 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.755641    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 03:19:58.804841 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.804818    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 03:19:58.856026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.855042    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b20027a-3439-4f84-8407-a9732ee5d748-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.856026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.855110    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6b20027a-3439-4f84-8407-a9732ee5d748-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.856026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.855140    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.856026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.855165    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.856026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.855204    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-config-volume\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.856026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.855238    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b20027a-3439-4f84-8407-a9732ee5d748-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.856026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.855287    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.856026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.855315    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b20027a-3439-4f84-8407-a9732ee5d748-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.856026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.855341    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b20027a-3439-4f84-8407-a9732ee5d748-config-out\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.856026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.855370    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txzbc\" (UniqueName: \"kubernetes.io/projected/6b20027a-3439-4f84-8407-a9732ee5d748-kube-api-access-txzbc\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.856026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.855430    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.856026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.855474    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-web-config\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.856026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.855506    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.857204 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.857159    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6b20027a-3439-4f84-8407-a9732ee5d748-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.860230 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.860199    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.860567 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.860546    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b20027a-3439-4f84-8407-a9732ee5d748-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.861513 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.861489    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b20027a-3439-4f84-8407-a9732ee5d748-config-out\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.861779 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.861762    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-config-volume\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.862395 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.862369    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b20027a-3439-4f84-8407-a9732ee5d748-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.862803 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.862784    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.866123 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.863800    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.866613 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.866553    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.866725 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.866710    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-web-config\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.867565 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.867363    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b20027a-3439-4f84-8407-a9732ee5d748-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.868783 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.868759    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.916906 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.916849    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-784c68f98c-wjj6h"]
Oct 31 03:19:58.919799 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.919782    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:58.926143 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.926124    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txzbc\" (UniqueName: \"kubernetes.io/projected/6b20027a-3439-4f84-8407-a9732ee5d748-kube-api-access-txzbc\") pod \"alertmanager-main-0\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:58.972499 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.972475    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 31 03:19:58.975163 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.975139    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 31 03:19:58.984014 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:58.983984    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-784c68f98c-wjj6h"]
Oct 31 03:19:59.058025 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.057994    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.058143 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.058047    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.058247 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.058225    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.058281 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.058259    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dw9v\" (UniqueName: \"kubernetes.io/projected/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-kube-api-access-7dw9v\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.058318 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.058279    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-secret-thanos-querier-tls\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.058318 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.058297    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-secret-grpc-tls\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.058398 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.058366    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-tmp\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.058503 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.058483    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-metrics-client-ca\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.058582 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.058542    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.073287 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.073228    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 31 03:19:59.073287 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.073228    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 31 03:19:59.073425 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.073233    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 31 03:19:59.073425 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.073252    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4l5i2156g7v73\""
Oct 31 03:19:59.073425 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.073324    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-82cj5\""
Oct 31 03:19:59.159046 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.158966    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-metrics-client-ca\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.159046 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.159009    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.159046 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.159030    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.159304 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.159050    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.159304 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.159105    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.159304 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.159125    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dw9v\" (UniqueName: \"kubernetes.io/projected/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-kube-api-access-7dw9v\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.159304 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.159143    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-secret-thanos-querier-tls\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.159304 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.159168    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-secret-grpc-tls\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.159304 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.159196    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-tmp\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.159797 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.159742    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-tmp\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.159797 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.159794    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-metrics-client-ca\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.161989 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.161958    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-secret-grpc-tls\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.162112 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.162092    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.162269 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.162253    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.162308 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.162276    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.162905 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.162858    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-secret-thanos-querier-tls\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.163117 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.163083    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.218862 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.218831    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:19:59.221575 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.221550    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dw9v\" (UniqueName: \"kubernetes.io/projected/a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac-kube-api-access-7dw9v\") pod \"thanos-querier-784c68f98c-wjj6h\" (UID: \"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac\") " pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.243260 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.243235    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:19:59.645677 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.645639    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 03:19:59.648294 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:59.648257    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b20027a_3439_4f84_8407_a9732ee5d748.slice/crio-65153e9e52c3dab58e21a11672e9d054b6fb5e80506a81be37b4933d1e9ef33b WatchSource:0}: Error finding container 65153e9e52c3dab58e21a11672e9d054b6fb5e80506a81be37b4933d1e9ef33b: Status 404 returned error can't find the container with id 65153e9e52c3dab58e21a11672e9d054b6fb5e80506a81be37b4933d1e9ef33b
Oct 31 03:19:59.725471 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.725440    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-784c68f98c-wjj6h"]
Oct 31 03:19:59.728687 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:19:59.728662    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d0bc3d_fc8f_4d56_a568_b2bb1b6b4aac.slice/crio-8dc7e521aec1c120ebe375828264cf2c3652f6571e5fc053f2e9de8a3f8bb69d WatchSource:0}: Error finding container 8dc7e521aec1c120ebe375828264cf2c3652f6571e5fc053f2e9de8a3f8bb69d: Status 404 returned error can't find the container with id 8dc7e521aec1c120ebe375828264cf2c3652f6571e5fc053f2e9de8a3f8bb69d
Oct 31 03:19:59.742311 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.742277    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb" event={"ID":"213aee5f-38c2-41ce-8790-e37ec9437e10","Type":"ContainerStarted","Data":"c9eb36447bd215f892174c7496fed274d712a5ccaa057d0546f111d125995406"}
Oct 31 03:19:59.742711 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.742336    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb" event={"ID":"213aee5f-38c2-41ce-8790-e37ec9437e10","Type":"ContainerStarted","Data":"50d1c8529a07996d182b86d6d970927ff977f01a12da62eef7c4783e4d946b74"}
Oct 31 03:19:59.742711 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.742367    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb" event={"ID":"213aee5f-38c2-41ce-8790-e37ec9437e10","Type":"ContainerStarted","Data":"93c5605afb3d126db8ad4b36845ccd067a3c7cf9ea3c70ac2b7a50b5fddbe476"}
Oct 31 03:19:59.743550 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.743528    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h" event={"ID":"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac","Type":"ContainerStarted","Data":"8dc7e521aec1c120ebe375828264cf2c3652f6571e5fc053f2e9de8a3f8bb69d"}
Oct 31 03:19:59.744803 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.744779    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b20027a-3439-4f84-8407-a9732ee5d748","Type":"ContainerStarted","Data":"65153e9e52c3dab58e21a11672e9d054b6fb5e80506a81be37b4933d1e9ef33b"}
Oct 31 03:19:59.746859 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.746830    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8gkd5" event={"ID":"311bf3de-3730-4e0b-9872-192c9bde06cf","Type":"ContainerStarted","Data":"f9302fa36d060fbff00d234d4f92136ca6a8ff5989c56d97ef8881df28a716f6"}
Oct 31 03:19:59.746969 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.746867    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8gkd5" event={"ID":"311bf3de-3730-4e0b-9872-192c9bde06cf","Type":"ContainerStarted","Data":"2525d9fa09b844d95e40aeb75faedf542aef7058241fda214dd953c21ec35832"}
Oct 31 03:19:59.861411 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.861364    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h4gsb" podStartSLOduration=2.043403588 podStartE2EDuration="3.861348548s" totalImagesPullingTime="1.81794496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:59.860708039 +0000 UTC m=+25.371690415" watchObservedRunningTime="2026-10-31 03:19:59.861348548 +0000 UTC m=+25.372330924"
Oct 31 03:19:59.980712 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:19:59.980654    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8gkd5" podStartSLOduration=2.571388992 podStartE2EDuration="3.9806369s" totalImagesPullingTime="1.335526488s" totalInitContainerRuntime="73.72142ms" isStatefulPod=true podCreationTimestamp="2026-10-31 03:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:59.979925452 +0000 UTC m=+25.490908155" watchObservedRunningTime="2026-10-31 03:19:59.9806369 +0000 UTC m=+25.491619277"
Oct 31 03:20:00.953939 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:00.953243    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-mtnb7"]
Oct 31 03:20:00.955620 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:00.955585    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mtnb7"
Oct 31 03:20:01.017537 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.017506    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-mtnb7"]
Oct 31 03:20:01.062458 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.062425    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 31 03:20:01.062458 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.062425    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dvnx6\""
Oct 31 03:20:01.076910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.076852    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca103aa-c5e7-4753-b429-159893db3ab6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-mtnb7\" (UID: \"3ca103aa-c5e7-4753-b429-159893db3ab6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mtnb7"
Oct 31 03:20:01.077083 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.076974    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x74n\" (UniqueName: \"kubernetes.io/projected/3ca103aa-c5e7-4753-b429-159893db3ab6-kube-api-access-9x74n\") pod \"monitoring-plugin-77f4f76594-mtnb7\" (UID: \"3ca103aa-c5e7-4753-b429-159893db3ab6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mtnb7"
Oct 31 03:20:01.178300 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.177519    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca103aa-c5e7-4753-b429-159893db3ab6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-mtnb7\" (UID: \"3ca103aa-c5e7-4753-b429-159893db3ab6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mtnb7"
Oct 31 03:20:01.178300 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.177572    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x74n\" (UniqueName: \"kubernetes.io/projected/3ca103aa-c5e7-4753-b429-159893db3ab6-kube-api-access-9x74n\") pod \"monitoring-plugin-77f4f76594-mtnb7\" (UID: \"3ca103aa-c5e7-4753-b429-159893db3ab6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mtnb7"
Oct 31 03:20:01.180835 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.180803    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca103aa-c5e7-4753-b429-159893db3ab6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-mtnb7\" (UID: \"3ca103aa-c5e7-4753-b429-159893db3ab6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mtnb7"
Oct 31 03:20:01.220963 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.219938    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-846c75f654-4rxqg"]
Oct 31 03:20:01.222290 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.222259    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:01.243134 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.243072    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x74n\" (UniqueName: \"kubernetes.io/projected/3ca103aa-c5e7-4753-b429-159893db3ab6-kube-api-access-9x74n\") pod \"monitoring-plugin-77f4f76594-mtnb7\" (UID: \"3ca103aa-c5e7-4753-b429-159893db3ab6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-mtnb7"
Oct 31 03:20:01.269033 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.269005    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mtnb7"
Oct 31 03:20:01.276859 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.275060    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2628g\""
Oct 31 03:20:01.276859 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.275095    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dnivo9uk5kru8\""
Oct 31 03:20:01.276859 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.275163    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 03:20:01.276859 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.275170    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 31 03:20:01.276859 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.275238    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 31 03:20:01.276859 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.275247    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 31 03:20:01.282803 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.282776    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-846c75f654-4rxqg"]
Oct 31 03:20:01.379830 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.379798    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2cbb3341-678d-484b-b2c1-2112c6ae4f34-audit-log\") pod \"metrics-server-846c75f654-4rxqg\" (UID: \"2cbb3341-678d-484b-b2c1-2112c6ae4f34\") " pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:01.379830 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.379830    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2cbb3341-678d-484b-b2c1-2112c6ae4f34-client-ca-bundle\") pod \"metrics-server-846c75f654-4rxqg\" (UID: \"2cbb3341-678d-484b-b2c1-2112c6ae4f34\") " pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:01.380100 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.379964    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cbb3341-678d-484b-b2c1-2112c6ae4f34-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-846c75f654-4rxqg\" (UID: \"2cbb3341-678d-484b-b2c1-2112c6ae4f34\") " pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:01.380100 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.380003    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2cbb3341-678d-484b-b2c1-2112c6ae4f34-secret-metrics-server-tls\") pod \"metrics-server-846c75f654-4rxqg\" (UID: \"2cbb3341-678d-484b-b2c1-2112c6ae4f34\") " pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:01.380100 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.380042    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2cbb3341-678d-484b-b2c1-2112c6ae4f34-metrics-server-audit-profiles\") pod \"metrics-server-846c75f654-4rxqg\" (UID: \"2cbb3341-678d-484b-b2c1-2112c6ae4f34\") " pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:01.380255 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.380170    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k442c\" (UniqueName: \"kubernetes.io/projected/2cbb3341-678d-484b-b2c1-2112c6ae4f34-kube-api-access-k442c\") pod \"metrics-server-846c75f654-4rxqg\" (UID: \"2cbb3341-678d-484b-b2c1-2112c6ae4f34\") " pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:01.380255 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.380214    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2cbb3341-678d-484b-b2c1-2112c6ae4f34-secret-metrics-server-client-certs\") pod \"metrics-server-846c75f654-4rxqg\" (UID: \"2cbb3341-678d-484b-b2c1-2112c6ae4f34\") " pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:01.481088 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.480997    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2cbb3341-678d-484b-b2c1-2112c6ae4f34-secret-metrics-server-tls\") pod \"metrics-server-846c75f654-4rxqg\" (UID: \"2cbb3341-678d-484b-b2c1-2112c6ae4f34\") " pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:01.481088 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.481052    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2cbb3341-678d-484b-b2c1-2112c6ae4f34-metrics-server-audit-profiles\") pod \"metrics-server-846c75f654-4rxqg\" (UID: \"2cbb3341-678d-484b-b2c1-2112c6ae4f34\") " pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:01.481296 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.481093    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k442c\" (UniqueName: \"kubernetes.io/projected/2cbb3341-678d-484b-b2c1-2112c6ae4f34-kube-api-access-k442c\") pod \"metrics-server-846c75f654-4rxqg\" (UID: \"2cbb3341-678d-484b-b2c1-2112c6ae4f34\") " pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:01.481296 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.481124    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2cbb3341-678d-484b-b2c1-2112c6ae4f34-secret-metrics-server-client-certs\") pod \"metrics-server-846c75f654-4rxqg\" (UID: \"2cbb3341-678d-484b-b2c1-2112c6ae4f34\") " pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:01.481296 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.481177    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2cbb3341-678d-484b-b2c1-2112c6ae4f34-audit-log\") pod \"metrics-server-846c75f654-4rxqg\" (UID: \"2cbb3341-678d-484b-b2c1-2112c6ae4f34\") " pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:01.481296 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.481196    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2cbb3341-678d-484b-b2c1-2112c6ae4f34-client-ca-bundle\") pod \"metrics-server-846c75f654-4rxqg\" (UID: \"2cbb3341-678d-484b-b2c1-2112c6ae4f34\") " pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:01.481296 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.481229    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cbb3341-678d-484b-b2c1-2112c6ae4f34-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-846c75f654-4rxqg\" (UID: \"2cbb3341-678d-484b-b2c1-2112c6ae4f34\") " pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:01.481653 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.481627    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2cbb3341-678d-484b-b2c1-2112c6ae4f34-audit-log\") pod \"metrics-server-846c75f654-4rxqg\" (UID: \"2cbb3341-678d-484b-b2c1-2112c6ae4f34\") " pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:01.481987 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.481963    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2cbb3341-678d-484b-b2c1-2112c6ae4f34-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-846c75f654-4rxqg\" (UID: \"2cbb3341-678d-484b-b2c1-2112c6ae4f34\") " pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:01.482199 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.482173    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2cbb3341-678d-484b-b2c1-2112c6ae4f34-metrics-server-audit-profiles\") pod \"metrics-server-846c75f654-4rxqg\" (UID: \"2cbb3341-678d-484b-b2c1-2112c6ae4f34\") " pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:01.483960 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.483940    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2cbb3341-678d-484b-b2c1-2112c6ae4f34-secret-metrics-server-tls\") pod \"metrics-server-846c75f654-4rxqg\" (UID: \"2cbb3341-678d-484b-b2c1-2112c6ae4f34\") " pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:01.484147 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.484127    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2cbb3341-678d-484b-b2c1-2112c6ae4f34-client-ca-bundle\") pod \"metrics-server-846c75f654-4rxqg\" (UID: \"2cbb3341-678d-484b-b2c1-2112c6ae4f34\") " pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:01.484267 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.484247    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2cbb3341-678d-484b-b2c1-2112c6ae4f34-secret-metrics-server-client-certs\") pod \"metrics-server-846c75f654-4rxqg\" (UID: \"2cbb3341-678d-484b-b2c1-2112c6ae4f34\") " pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:01.543084 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.543043    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k442c\" (UniqueName: \"kubernetes.io/projected/2cbb3341-678d-484b-b2c1-2112c6ae4f34-kube-api-access-k442c\") pod \"metrics-server-846c75f654-4rxqg\" (UID: \"2cbb3341-678d-484b-b2c1-2112c6ae4f34\") " pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:01.649181 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.649151    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-mtnb7"]
Oct 31 03:20:01.652143 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:20:01.652104    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca103aa_c5e7_4753_b429_159893db3ab6.slice/crio-7d82f5a83e2bfb83be8cb8c57ee70ada3bce07b72db89b77de2fe17f92262cb6 WatchSource:0}: Error finding container 7d82f5a83e2bfb83be8cb8c57ee70ada3bce07b72db89b77de2fe17f92262cb6: Status 404 returned error can't find the container with id 7d82f5a83e2bfb83be8cb8c57ee70ada3bce07b72db89b77de2fe17f92262cb6
Oct 31 03:20:01.757390 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.757316    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mtnb7" event={"ID":"3ca103aa-c5e7-4753-b429-159893db3ab6","Type":"ContainerStarted","Data":"7d82f5a83e2bfb83be8cb8c57ee70ada3bce07b72db89b77de2fe17f92262cb6"}
Oct 31 03:20:01.759172 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.759147    4447 generic.go:356] "Generic (PLEG): container finished" podID="6b20027a-3439-4f84-8407-a9732ee5d748" containerID="8f5ebe275607e8aa7163bc912eeb1f7f24072d3014eb67faf25e5e28d90263e0" exitCode=0
Oct 31 03:20:01.759298 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.759209    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b20027a-3439-4f84-8407-a9732ee5d748","Type":"ContainerDied","Data":"8f5ebe275607e8aa7163bc912eeb1f7f24072d3014eb67faf25e5e28d90263e0"}
Oct 31 03:20:01.834527 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.834492    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:01.946577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.946540    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-666c5cbddb-69wsv"]
Oct 31 03:20:01.950867 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:01.950818    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.003952 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.003857    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 31 03:20:02.003952 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.003916    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 31 03:20:02.003952 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.003918    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 31 03:20:02.004697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.004131    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-5qsfp\""
Oct 31 03:20:02.004697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.004134    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 31 03:20:02.004697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.004193    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 31 03:20:02.054373 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.054281    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 31 03:20:02.055339 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.055311    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-666c5cbddb-69wsv"]
Oct 31 03:20:02.086123 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.086087    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/66185435-76f9-4541-b002-0f3f7bf43f52-telemeter-client-tls\") pod \"telemeter-client-666c5cbddb-69wsv\" (UID: \"66185435-76f9-4541-b002-0f3f7bf43f52\") " pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.086295 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.086135    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66185435-76f9-4541-b002-0f3f7bf43f52-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-666c5cbddb-69wsv\" (UID: \"66185435-76f9-4541-b002-0f3f7bf43f52\") " pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.086295 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.086159    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66185435-76f9-4541-b002-0f3f7bf43f52-telemeter-trusted-ca-bundle\") pod \"telemeter-client-666c5cbddb-69wsv\" (UID: \"66185435-76f9-4541-b002-0f3f7bf43f52\") " pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.086391 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.086299    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/66185435-76f9-4541-b002-0f3f7bf43f52-secret-telemeter-client\") pod \"telemeter-client-666c5cbddb-69wsv\" (UID: \"66185435-76f9-4541-b002-0f3f7bf43f52\") " pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.086391 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.086369    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66185435-76f9-4541-b002-0f3f7bf43f52-metrics-client-ca\") pod \"telemeter-client-666c5cbddb-69wsv\" (UID: \"66185435-76f9-4541-b002-0f3f7bf43f52\") " pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.086471 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.086428    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66185435-76f9-4541-b002-0f3f7bf43f52-serving-certs-ca-bundle\") pod \"telemeter-client-666c5cbddb-69wsv\" (UID: \"66185435-76f9-4541-b002-0f3f7bf43f52\") " pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.086523 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.086501    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/66185435-76f9-4541-b002-0f3f7bf43f52-federate-client-tls\") pod \"telemeter-client-666c5cbddb-69wsv\" (UID: \"66185435-76f9-4541-b002-0f3f7bf43f52\") " pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.086567 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.086546    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkfb2\" (UniqueName: \"kubernetes.io/projected/66185435-76f9-4541-b002-0f3f7bf43f52-kube-api-access-wkfb2\") pod \"telemeter-client-666c5cbddb-69wsv\" (UID: \"66185435-76f9-4541-b002-0f3f7bf43f52\") " pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.188061 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.188011    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66185435-76f9-4541-b002-0f3f7bf43f52-serving-certs-ca-bundle\") pod \"telemeter-client-666c5cbddb-69wsv\" (UID: \"66185435-76f9-4541-b002-0f3f7bf43f52\") " pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.188229 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.188073    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/66185435-76f9-4541-b002-0f3f7bf43f52-federate-client-tls\") pod \"telemeter-client-666c5cbddb-69wsv\" (UID: \"66185435-76f9-4541-b002-0f3f7bf43f52\") " pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.188294 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.188249    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkfb2\" (UniqueName: \"kubernetes.io/projected/66185435-76f9-4541-b002-0f3f7bf43f52-kube-api-access-wkfb2\") pod \"telemeter-client-666c5cbddb-69wsv\" (UID: \"66185435-76f9-4541-b002-0f3f7bf43f52\") " pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.188348 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.188303    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/66185435-76f9-4541-b002-0f3f7bf43f52-telemeter-client-tls\") pod \"telemeter-client-666c5cbddb-69wsv\" (UID: \"66185435-76f9-4541-b002-0f3f7bf43f52\") " pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.188348 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.188335    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66185435-76f9-4541-b002-0f3f7bf43f52-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-666c5cbddb-69wsv\" (UID: \"66185435-76f9-4541-b002-0f3f7bf43f52\") " pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.188451 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.188353    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66185435-76f9-4541-b002-0f3f7bf43f52-telemeter-trusted-ca-bundle\") pod \"telemeter-client-666c5cbddb-69wsv\" (UID: \"66185435-76f9-4541-b002-0f3f7bf43f52\") " pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.188451 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.188389    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/66185435-76f9-4541-b002-0f3f7bf43f52-secret-telemeter-client\") pod \"telemeter-client-666c5cbddb-69wsv\" (UID: \"66185435-76f9-4541-b002-0f3f7bf43f52\") " pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.188451 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.188433    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66185435-76f9-4541-b002-0f3f7bf43f52-metrics-client-ca\") pod \"telemeter-client-666c5cbddb-69wsv\" (UID: \"66185435-76f9-4541-b002-0f3f7bf43f52\") " pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.188842 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.188811    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66185435-76f9-4541-b002-0f3f7bf43f52-serving-certs-ca-bundle\") pod \"telemeter-client-666c5cbddb-69wsv\" (UID: \"66185435-76f9-4541-b002-0f3f7bf43f52\") " pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.189003 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.188982    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66185435-76f9-4541-b002-0f3f7bf43f52-metrics-client-ca\") pod \"telemeter-client-666c5cbddb-69wsv\" (UID: \"66185435-76f9-4541-b002-0f3f7bf43f52\") " pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.193923 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.191319    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66185435-76f9-4541-b002-0f3f7bf43f52-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-666c5cbddb-69wsv\" (UID: \"66185435-76f9-4541-b002-0f3f7bf43f52\") " pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.193923 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.192344    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66185435-76f9-4541-b002-0f3f7bf43f52-telemeter-trusted-ca-bundle\") pod \"telemeter-client-666c5cbddb-69wsv\" (UID: \"66185435-76f9-4541-b002-0f3f7bf43f52\") " pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.193923 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.193392    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/66185435-76f9-4541-b002-0f3f7bf43f52-federate-client-tls\") pod \"telemeter-client-666c5cbddb-69wsv\" (UID: \"66185435-76f9-4541-b002-0f3f7bf43f52\") " pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.193923 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.193575    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/66185435-76f9-4541-b002-0f3f7bf43f52-telemeter-client-tls\") pod \"telemeter-client-666c5cbddb-69wsv\" (UID: \"66185435-76f9-4541-b002-0f3f7bf43f52\") " pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.197279 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.197256    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/66185435-76f9-4541-b002-0f3f7bf43f52-secret-telemeter-client\") pod \"telemeter-client-666c5cbddb-69wsv\" (UID: \"66185435-76f9-4541-b002-0f3f7bf43f52\") " pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.213111 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.213078    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-846c75f654-4rxqg"]
Oct 31 03:20:02.216338 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:20:02.216314    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cbb3341_678d_484b_b2c1_2112c6ae4f34.slice/crio-9dd7f11e43eb9236b66b948a9d1baacaa3c386affa7eb7313568d7898d8319ee WatchSource:0}: Error finding container 9dd7f11e43eb9236b66b948a9d1baacaa3c386affa7eb7313568d7898d8319ee: Status 404 returned error can't find the container with id 9dd7f11e43eb9236b66b948a9d1baacaa3c386affa7eb7313568d7898d8319ee
Oct 31 03:20:02.252602 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.252567    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkfb2\" (UniqueName: \"kubernetes.io/projected/66185435-76f9-4541-b002-0f3f7bf43f52-kube-api-access-wkfb2\") pod \"telemeter-client-666c5cbddb-69wsv\" (UID: \"66185435-76f9-4541-b002-0f3f7bf43f52\") " pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.268293 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.268254    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv"
Oct 31 03:20:02.712312 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.712284    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-666c5cbddb-69wsv"]
Oct 31 03:20:02.715349 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:20:02.715250    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66185435_76f9_4541_b002_0f3f7bf43f52.slice/crio-b0a413929368006dd692900396efb4955ca3c298f016455b1f8884c979e0cbea WatchSource:0}: Error finding container b0a413929368006dd692900396efb4955ca3c298f016455b1f8884c979e0cbea: Status 404 returned error can't find the container with id b0a413929368006dd692900396efb4955ca3c298f016455b1f8884c979e0cbea
Oct 31 03:20:02.764636 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.764596    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h" event={"ID":"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac","Type":"ContainerStarted","Data":"477edfd92b4805d9ad3c62898dfbd1fbec08f04b3a5354cb48742ccc6c7045ef"}
Oct 31 03:20:02.765643 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.765619    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv" event={"ID":"66185435-76f9-4541-b002-0f3f7bf43f52","Type":"ContainerStarted","Data":"b0a413929368006dd692900396efb4955ca3c298f016455b1f8884c979e0cbea"}
Oct 31 03:20:02.766756 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:02.766733    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-846c75f654-4rxqg" event={"ID":"2cbb3341-678d-484b-b2c1-2112c6ae4f34","Type":"ContainerStarted","Data":"9dd7f11e43eb9236b66b948a9d1baacaa3c386affa7eb7313568d7898d8319ee"}
Oct 31 03:20:03.773879 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:03.773845    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h" event={"ID":"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac","Type":"ContainerStarted","Data":"868b8519a60c3f87c8b0b7b85a94463cd23425facf75d25cf43de4ac0f4ddb3e"}
Oct 31 03:20:03.774371 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:03.773946    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h" event={"ID":"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac","Type":"ContainerStarted","Data":"74b908662bdcb9de1a60fbc078623d1b4bb4f75ab0f3cbaf289b9b21a7dc7905"}
Oct 31 03:20:05.787704 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:05.787671    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b20027a-3439-4f84-8407-a9732ee5d748","Type":"ContainerStarted","Data":"75066256a247ed923a17f85af18049d413100cb4f3def0dd70c863b8290e4616"}
Oct 31 03:20:05.788046 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:05.787806    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b20027a-3439-4f84-8407-a9732ee5d748","Type":"ContainerStarted","Data":"4a096211e692a0e95f9e56f42920fb738be670ece24bcb9336a352aaf68bcc4b"}
Oct 31 03:20:05.788046 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:05.787864    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b20027a-3439-4f84-8407-a9732ee5d748","Type":"ContainerStarted","Data":"1ee61aa324292d914d10b5cf9540aa3b241e47c439c714606796bb5f6138aa6e"}
Oct 31 03:20:05.788046 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:05.787940    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b20027a-3439-4f84-8407-a9732ee5d748","Type":"ContainerStarted","Data":"a2064bc248b06923e538581bf7d24685d8eeb07727d2ff69dda424d420fc8383"}
Oct 31 03:20:05.788046 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:05.787991    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b20027a-3439-4f84-8407-a9732ee5d748","Type":"ContainerStarted","Data":"38da9ef6df6f0a3be404101c143951654ffb8078e6f2f2cbe01e92c4ae037a75"}
Oct 31 03:20:05.788163 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:05.788028    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b20027a-3439-4f84-8407-a9732ee5d748","Type":"ContainerStarted","Data":"e1034fa2c52e06903051a4b34ad0de7873bcfd4d984e7a89f7d28ef9ae3ff24f"}
Oct 31 03:20:05.791366 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:05.791344    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h" event={"ID":"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac","Type":"ContainerStarted","Data":"82ae31c6c163ef93eb621b6241602cbf433e935acb2d6a3c10839114ded78c6b"}
Oct 31 03:20:05.791799 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:05.791693    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h" event={"ID":"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac","Type":"ContainerStarted","Data":"bcdb46ae19cc6853c936062250e103cb8da2301113783f36056dc52ba3225463"}
Oct 31 03:20:05.792025 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:05.792007    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h" event={"ID":"a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac","Type":"ContainerStarted","Data":"9f58025a34c5a0672175e547f4ffe9df93f10f0c3b0196ec605e7937a98e0179"}
Oct 31 03:20:05.792236 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:05.792226    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:20:05.794382 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:05.794362    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mtnb7" event={"ID":"3ca103aa-c5e7-4753-b429-159893db3ab6","Type":"ContainerStarted","Data":"2c869ae7a23732f2301f727f64147176a1e5fdd6d3a95a21cdf6acaebd7fc02d"}
Oct 31 03:20:05.795184 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:05.794943    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mtnb7"
Oct 31 03:20:05.797549 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:05.797347    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv" event={"ID":"66185435-76f9-4541-b002-0f3f7bf43f52","Type":"ContainerStarted","Data":"963733079da4e968e4547bc390bb7ddc0e2589e9922c70b0888f32490002ed7d"}
Oct 31 03:20:05.799906 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:05.799862    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-846c75f654-4rxqg" event={"ID":"2cbb3341-678d-484b-b2c1-2112c6ae4f34","Type":"ContainerStarted","Data":"9c5cc69a2c42490b32f96e16bcee328aa97f12724f0127c0d6b12316dcc3ee67"}
Oct 31 03:20:05.803641 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:05.803622    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mtnb7"
Oct 31 03:20:05.976808 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:05.976763    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.326902037 podStartE2EDuration="7.976747906s" totalImagesPullingTime="4.639994609s" totalInitContainerRuntime="9.85126ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:05.975792733 +0000 UTC m=+31.486775112" watchObservedRunningTime="2026-10-31 03:20:05.976747906 +0000 UTC m=+31.487730277"
Oct 31 03:20:06.101225 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:06.101173    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h" podStartSLOduration=2.643866111 podStartE2EDuration="8.101157859s" totalImagesPullingTime="5.457291748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:06.100478631 +0000 UTC m=+31.611461010" watchObservedRunningTime="2026-10-31 03:20:06.101157859 +0000 UTC m=+31.612140235"
Oct 31 03:20:06.216691 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:06.216627    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-mtnb7" podStartSLOduration=3.072035741 podStartE2EDuration="6.216610967s" totalImagesPullingTime="3.144575226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:06.215903331 +0000 UTC m=+31.726885706" watchObservedRunningTime="2026-10-31 03:20:06.216610967 +0000 UTC m=+31.727593343"
Oct 31 03:20:06.340700 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:06.340633    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-846c75f654-4rxqg" podStartSLOduration=2.749211711 podStartE2EDuration="5.340614477s" totalImagesPullingTime="2.591402766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:06.33917003 +0000 UTC m=+31.850152406" watchObservedRunningTime="2026-10-31 03:20:06.340614477 +0000 UTC m=+31.851596886"
Oct 31 03:20:06.806160 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:06.806125    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv" event={"ID":"66185435-76f9-4541-b002-0f3f7bf43f52","Type":"ContainerStarted","Data":"22e2ca6f1afab8bc75c4b0f89b011bd1e2b9b0e5a54edbec5c99d8182484fef7"}
Oct 31 03:20:06.806654 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:06.806200    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv" event={"ID":"66185435-76f9-4541-b002-0f3f7bf43f52","Type":"ContainerStarted","Data":"9599573c59ccee6c55a6598a2a5c135faa9601e485086b07701c55b1a64f00ff"}
Oct 31 03:20:06.929964 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:06.929916    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-666c5cbddb-69wsv" podStartSLOduration=2.958444857 podStartE2EDuration="5.929871994s" totalImagesPullingTime="2.971427137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:06.929182314 +0000 UTC m=+32.440164693" watchObservedRunningTime="2026-10-31 03:20:06.929871994 +0000 UTC m=+32.440854371"
Oct 31 03:20:07.463273 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:07.463241    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rtp6w"
Oct 31 03:20:07.654656 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:07.654630    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-hc4wm"]
Oct 31 03:20:07.674777 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:07.674752    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-hc4wm"
Oct 31 03:20:07.717678 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:07.717603    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-hc4wm"]
Oct 31 03:20:07.727203 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:07.727169    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-f97dp\""
Oct 31 03:20:07.727203 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:07.727190    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 03:20:07.727429 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:07.727262    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 03:20:07.818442 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:07.818417    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-784c68f98c-wjj6h"
Oct 31 03:20:07.839422 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:07.839397    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqprv\" (UniqueName: \"kubernetes.io/projected/76279c27-5b93-4ca2-98c3-cfd9d5a651ad-kube-api-access-hqprv\") pod \"downloads-6dd789c8b7-hc4wm\" (UID: \"76279c27-5b93-4ca2-98c3-cfd9d5a651ad\") " pod="openshift-console/downloads-6dd789c8b7-hc4wm"
Oct 31 03:20:07.839600 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:07.839435    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76279c27-5b93-4ca2-98c3-cfd9d5a651ad-tmp\") pod \"downloads-6dd789c8b7-hc4wm\" (UID: \"76279c27-5b93-4ca2-98c3-cfd9d5a651ad\") " pod="openshift-console/downloads-6dd789c8b7-hc4wm"
Oct 31 03:20:07.940655 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:07.940609    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqprv\" (UniqueName: \"kubernetes.io/projected/76279c27-5b93-4ca2-98c3-cfd9d5a651ad-kube-api-access-hqprv\") pod \"downloads-6dd789c8b7-hc4wm\" (UID: \"76279c27-5b93-4ca2-98c3-cfd9d5a651ad\") " pod="openshift-console/downloads-6dd789c8b7-hc4wm"
Oct 31 03:20:07.940847 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:07.940671    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76279c27-5b93-4ca2-98c3-cfd9d5a651ad-tmp\") pod \"downloads-6dd789c8b7-hc4wm\" (UID: \"76279c27-5b93-4ca2-98c3-cfd9d5a651ad\") " pod="openshift-console/downloads-6dd789c8b7-hc4wm"
Oct 31 03:20:07.941073 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:07.941056    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76279c27-5b93-4ca2-98c3-cfd9d5a651ad-tmp\") pod \"downloads-6dd789c8b7-hc4wm\" (UID: \"76279c27-5b93-4ca2-98c3-cfd9d5a651ad\") " pod="openshift-console/downloads-6dd789c8b7-hc4wm"
Oct 31 03:20:08.002782 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:08.002705    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqprv\" (UniqueName: \"kubernetes.io/projected/76279c27-5b93-4ca2-98c3-cfd9d5a651ad-kube-api-access-hqprv\") pod \"downloads-6dd789c8b7-hc4wm\" (UID: \"76279c27-5b93-4ca2-98c3-cfd9d5a651ad\") " pod="openshift-console/downloads-6dd789c8b7-hc4wm"
Oct 31 03:20:08.299325 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:08.299243    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-hc4wm"
Oct 31 03:20:08.784966 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:08.784931    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-hc4wm"]
Oct 31 03:20:08.828252 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:20:08.828215    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76279c27_5b93_4ca2_98c3_cfd9d5a651ad.slice/crio-25be542fc106fd15e2e51e9447961b5efec1b25b86e00f491b19d20e9a69e705 WatchSource:0}: Error finding container 25be542fc106fd15e2e51e9447961b5efec1b25b86e00f491b19d20e9a69e705: Status 404 returned error can't find the container with id 25be542fc106fd15e2e51e9447961b5efec1b25b86e00f491b19d20e9a69e705
Oct 31 03:20:08.833930 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:08.833905    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c656c8f7d-cprkw"]
Oct 31 03:20:08.864299 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:08.864271    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:08.890941 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:08.890903    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c656c8f7d-cprkw"]
Oct 31 03:20:08.917077 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:08.917039    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 03:20:08.917258 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:08.917200    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6rqql\""
Oct 31 03:20:08.917258 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:08.917200    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 03:20:08.948433 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:08.948401    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 03:20:08.948433 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:08.948401    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 03:20:08.948618 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:08.948440    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 03:20:08.948618 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:08.948569    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b7b96f3-2670-4021-a387-15f089d2f1d1-oauth-serving-cert\") pod \"console-7c656c8f7d-cprkw\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") " pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:08.948734 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:08.948703    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b7b96f3-2670-4021-a387-15f089d2f1d1-console-config\") pod \"console-7c656c8f7d-cprkw\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") " pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:08.949076 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:08.948796    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b7b96f3-2670-4021-a387-15f089d2f1d1-tmp\") pod \"console-7c656c8f7d-cprkw\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") " pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:08.949076 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:08.948864    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b7b96f3-2670-4021-a387-15f089d2f1d1-console-oauth-config\") pod \"console-7c656c8f7d-cprkw\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") " pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:08.949076 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:08.948978    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b7b96f3-2670-4021-a387-15f089d2f1d1-console-serving-cert\") pod \"console-7c656c8f7d-cprkw\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") " pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:08.949076 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:08.949046    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b7b96f3-2670-4021-a387-15f089d2f1d1-service-ca\") pod \"console-7c656c8f7d-cprkw\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") " pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:08.949426 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:08.949398    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrdb2\" (UniqueName: \"kubernetes.io/projected/7b7b96f3-2670-4021-a387-15f089d2f1d1-kube-api-access-zrdb2\") pod \"console-7c656c8f7d-cprkw\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") " pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:09.050993 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:09.050906    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b7b96f3-2670-4021-a387-15f089d2f1d1-console-serving-cert\") pod \"console-7c656c8f7d-cprkw\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") " pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:09.050993 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:09.050950    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b7b96f3-2670-4021-a387-15f089d2f1d1-service-ca\") pod \"console-7c656c8f7d-cprkw\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") " pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:09.050993 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:09.050972    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrdb2\" (UniqueName: \"kubernetes.io/projected/7b7b96f3-2670-4021-a387-15f089d2f1d1-kube-api-access-zrdb2\") pod \"console-7c656c8f7d-cprkw\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") " pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:09.051256 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:09.051014    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b7b96f3-2670-4021-a387-15f089d2f1d1-oauth-serving-cert\") pod \"console-7c656c8f7d-cprkw\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") " pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:09.051256 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:09.051063    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b7b96f3-2670-4021-a387-15f089d2f1d1-console-config\") pod \"console-7c656c8f7d-cprkw\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") " pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:09.051256 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:09.051092    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b7b96f3-2670-4021-a387-15f089d2f1d1-tmp\") pod \"console-7c656c8f7d-cprkw\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") " pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:09.051256 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:09.051121    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b7b96f3-2670-4021-a387-15f089d2f1d1-console-oauth-config\") pod \"console-7c656c8f7d-cprkw\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") " pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:09.051514 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:09.051492    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b7b96f3-2670-4021-a387-15f089d2f1d1-tmp\") pod \"console-7c656c8f7d-cprkw\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") " pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:09.051753 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:09.051732    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b7b96f3-2670-4021-a387-15f089d2f1d1-service-ca\") pod \"console-7c656c8f7d-cprkw\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") " pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:09.051836 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:09.051772    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b7b96f3-2670-4021-a387-15f089d2f1d1-oauth-serving-cert\") pod \"console-7c656c8f7d-cprkw\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") " pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:09.051836 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:09.051772    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b7b96f3-2670-4021-a387-15f089d2f1d1-console-config\") pod \"console-7c656c8f7d-cprkw\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") " pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:09.053699 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:09.053676    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b7b96f3-2670-4021-a387-15f089d2f1d1-console-oauth-config\") pod \"console-7c656c8f7d-cprkw\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") " pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:09.053699 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:09.053692    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b7b96f3-2670-4021-a387-15f089d2f1d1-console-serving-cert\") pod \"console-7c656c8f7d-cprkw\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") " pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:09.112873 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:09.112843    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrdb2\" (UniqueName: \"kubernetes.io/projected/7b7b96f3-2670-4021-a387-15f089d2f1d1-kube-api-access-zrdb2\") pod \"console-7c656c8f7d-cprkw\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") " pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:09.187044 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:09.187009    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:09.586137 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:09.586101    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c656c8f7d-cprkw"]
Oct 31 03:20:09.589029 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:20:09.589002    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b7b96f3_2670_4021_a387_15f089d2f1d1.slice/crio-79f87b5cb777da675e6da0a320566c379c9bf67aeac22b5a997c27ad2a5d47da WatchSource:0}: Error finding container 79f87b5cb777da675e6da0a320566c379c9bf67aeac22b5a997c27ad2a5d47da: Status 404 returned error can't find the container with id 79f87b5cb777da675e6da0a320566c379c9bf67aeac22b5a997c27ad2a5d47da
Oct 31 03:20:09.822061 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:09.822024    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-hc4wm" event={"ID":"76279c27-5b93-4ca2-98c3-cfd9d5a651ad","Type":"ContainerStarted","Data":"25be542fc106fd15e2e51e9447961b5efec1b25b86e00f491b19d20e9a69e705"}
Oct 31 03:20:09.823432 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:09.823408    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c656c8f7d-cprkw" event={"ID":"7b7b96f3-2670-4021-a387-15f089d2f1d1","Type":"ContainerStarted","Data":"79f87b5cb777da675e6da0a320566c379c9bf67aeac22b5a997c27ad2a5d47da"}
Oct 31 03:20:13.847717 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:13.847684    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c656c8f7d-cprkw" event={"ID":"7b7b96f3-2670-4021-a387-15f089d2f1d1","Type":"ContainerStarted","Data":"f385bd1a3bf4f256d99c589caaf31dfac466d4112ed8fbe2f601f8840e10e2c3"}
Oct 31 03:20:13.968433 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:13.968375    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7c656c8f7d-cprkw" podStartSLOduration=2.638859104 podStartE2EDuration="5.968355392s" totalImagesPullingTime="3.329496288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:13.967658375 +0000 UTC m=+39.478640749" watchObservedRunningTime="2026-10-31 03:20:13.968355392 +0000 UTC m=+39.479337842"
Oct 31 03:20:19.003006 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:20:19.002927    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:20:19.188189 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:19.188151    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:19.188377 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:19.188225    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:19.193272 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:19.193252    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:19.878374 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:19.878347    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:21.835560 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:21.835528    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:21.836026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:21.835624    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:22.355205 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.355167    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d99d6f6d8-n4w55"]
Oct 31 03:20:22.359927 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.359901    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.368726 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.368702    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64782f21-9364-4cd2-9731-cd27f2ba2c29-console-oauth-config\") pod \"console-6d99d6f6d8-n4w55\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") " pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.368860 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.368730    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64782f21-9364-4cd2-9731-cd27f2ba2c29-tmp\") pod \"console-6d99d6f6d8-n4w55\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") " pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.368860 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.368748    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-oauth-serving-cert\") pod \"console-6d99d6f6d8-n4w55\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") " pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.368860 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.368768    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5tcb\" (UniqueName: \"kubernetes.io/projected/64782f21-9364-4cd2-9731-cd27f2ba2c29-kube-api-access-r5tcb\") pod \"console-6d99d6f6d8-n4w55\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") " pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.368860 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.368852    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-console-config\") pod \"console-6d99d6f6d8-n4w55\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") " pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.369098 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.368961    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-trusted-ca-bundle\") pod \"console-6d99d6f6d8-n4w55\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") " pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.369098 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.369000    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-service-ca\") pod \"console-6d99d6f6d8-n4w55\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") " pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.369098 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.369042    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64782f21-9364-4cd2-9731-cd27f2ba2c29-console-serving-cert\") pod \"console-6d99d6f6d8-n4w55\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") " pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.469483 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.469448    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-trusted-ca-bundle\") pod \"console-6d99d6f6d8-n4w55\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") " pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.469647 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.469498    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-service-ca\") pod \"console-6d99d6f6d8-n4w55\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") " pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.469647 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.469528    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64782f21-9364-4cd2-9731-cd27f2ba2c29-console-serving-cert\") pod \"console-6d99d6f6d8-n4w55\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") " pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.469647 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.469565    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64782f21-9364-4cd2-9731-cd27f2ba2c29-console-oauth-config\") pod \"console-6d99d6f6d8-n4w55\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") " pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.469647 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.469585    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64782f21-9364-4cd2-9731-cd27f2ba2c29-tmp\") pod \"console-6d99d6f6d8-n4w55\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") " pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.469647 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.469601    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-oauth-serving-cert\") pod \"console-6d99d6f6d8-n4w55\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") " pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.469647 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.469620    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5tcb\" (UniqueName: \"kubernetes.io/projected/64782f21-9364-4cd2-9731-cd27f2ba2c29-kube-api-access-r5tcb\") pod \"console-6d99d6f6d8-n4w55\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") " pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.469647 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.469644    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-console-config\") pod \"console-6d99d6f6d8-n4w55\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") " pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.470610 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.470575    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-console-config\") pod \"console-6d99d6f6d8-n4w55\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") " pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.471161 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.471133    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64782f21-9364-4cd2-9731-cd27f2ba2c29-tmp\") pod \"console-6d99d6f6d8-n4w55\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") " pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.471309 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.471292    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-service-ca\") pod \"console-6d99d6f6d8-n4w55\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") " pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.471378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.471307    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-oauth-serving-cert\") pod \"console-6d99d6f6d8-n4w55\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") " pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.478448 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.478418    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64782f21-9364-4cd2-9731-cd27f2ba2c29-console-oauth-config\") pod \"console-6d99d6f6d8-n4w55\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") " pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.479468 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.479445    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64782f21-9364-4cd2-9731-cd27f2ba2c29-console-serving-cert\") pod \"console-6d99d6f6d8-n4w55\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") " pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.562900 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.562850    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 31 03:20:22.564431 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.564397    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d99d6f6d8-n4w55"]
Oct 31 03:20:22.566238 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.566218    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5tcb\" (UniqueName: \"kubernetes.io/projected/64782f21-9364-4cd2-9731-cd27f2ba2c29-kube-api-access-r5tcb\") pod \"console-6d99d6f6d8-n4w55\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") " pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.572544 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.572523    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-trusted-ca-bundle\") pod \"console-6d99d6f6d8-n4w55\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") " pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:22.673455 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:22.673371    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:23.054114 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:23.054084    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d99d6f6d8-n4w55"]
Oct 31 03:20:23.057451 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:20:23.057422    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64782f21_9364_4cd2_9731_cd27f2ba2c29.slice/crio-59f1aafad753210a9cf82e490eace13345db812ebd0e16fae786b22a4813fe12 WatchSource:0}: Error finding container 59f1aafad753210a9cf82e490eace13345db812ebd0e16fae786b22a4813fe12: Status 404 returned error can't find the container with id 59f1aafad753210a9cf82e490eace13345db812ebd0e16fae786b22a4813fe12
Oct 31 03:20:23.892431 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:23.892395    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d99d6f6d8-n4w55" event={"ID":"64782f21-9364-4cd2-9731-cd27f2ba2c29","Type":"ContainerStarted","Data":"b46aa23029c3551f5b3ed990bff2aef96a5f7291c4aff5b3ec8bea0c6cba96b0"}
Oct 31 03:20:23.892614 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:23.892457    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d99d6f6d8-n4w55" event={"ID":"64782f21-9364-4cd2-9731-cd27f2ba2c29","Type":"ContainerStarted","Data":"59f1aafad753210a9cf82e490eace13345db812ebd0e16fae786b22a4813fe12"}
Oct 31 03:20:24.011720 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:24.011664    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6d99d6f6d8-n4w55" podStartSLOduration=2.011646254 podStartE2EDuration="2.011646254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:24.010430003 +0000 UTC m=+49.521412381" watchObservedRunningTime="2026-10-31 03:20:24.011646254 +0000 UTC m=+49.522628631"
Oct 31 03:20:24.907081 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:24.907026    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-hc4wm" event={"ID":"76279c27-5b93-4ca2-98c3-cfd9d5a651ad","Type":"ContainerStarted","Data":"6b71060e4050b842600ba46b05c5861c8535764403f31cc698108358388c75c1"}
Oct 31 03:20:24.907603 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:24.907334    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-hc4wm"
Oct 31 03:20:24.908968 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:24.908928    4447 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-hc4wm container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.17:8080/\": dial tcp 10.134.0.17:8080: connect: connection refused" start-of-body=
Oct 31 03:20:24.909088 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:24.908990    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-hc4wm" podUID="76279c27-5b93-4ca2-98c3-cfd9d5a651ad" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.17:8080/\": dial tcp 10.134.0.17:8080: connect: connection refused"
Oct 31 03:20:25.125771 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:25.125651    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-hc4wm" podStartSLOduration=2.235373408 podStartE2EDuration="18.125635634s" totalImagesPullingTime="15.890262226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:25.086000552 +0000 UTC m=+50.596982930" watchObservedRunningTime="2026-10-31 03:20:25.125635634 +0000 UTC m=+50.636618015"
Oct 31 03:20:25.926818 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:25.926781    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-hc4wm"
Oct 31 03:20:32.674454 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:32.674421    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:32.675043 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:32.674766    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:32.679959 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:32.679935    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:32.946729 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:32.946665    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:20:33.097937 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:33.097875    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c656c8f7d-cprkw"]
Oct 31 03:20:41.839947 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:41.839909    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:41.843787 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:41.843768    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-846c75f654-4rxqg"
Oct 31 03:20:47.927072 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:47.927041    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-98f98995-rfsnl"]
Oct 31 03:20:47.948692 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:47.948659    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.031224 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.031187    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-98f98995-rfsnl"]
Oct 31 03:20:48.077192 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.077160    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17586315-565c-4d7b-8f72-3bda38ccbe83-tmp\") pod \"console-98f98995-rfsnl\" (UID: \"17586315-565c-4d7b-8f72-3bda38ccbe83\") " pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.077192 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.077200    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17586315-565c-4d7b-8f72-3bda38ccbe83-service-ca\") pod \"console-98f98995-rfsnl\" (UID: \"17586315-565c-4d7b-8f72-3bda38ccbe83\") " pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.077494 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.077215    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17586315-565c-4d7b-8f72-3bda38ccbe83-console-config\") pod \"console-98f98995-rfsnl\" (UID: \"17586315-565c-4d7b-8f72-3bda38ccbe83\") " pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.077494 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.077304    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17586315-565c-4d7b-8f72-3bda38ccbe83-console-oauth-config\") pod \"console-98f98995-rfsnl\" (UID: \"17586315-565c-4d7b-8f72-3bda38ccbe83\") " pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.077494 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.077381    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17586315-565c-4d7b-8f72-3bda38ccbe83-trusted-ca-bundle\") pod \"console-98f98995-rfsnl\" (UID: \"17586315-565c-4d7b-8f72-3bda38ccbe83\") " pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.077494 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.077414    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17586315-565c-4d7b-8f72-3bda38ccbe83-console-serving-cert\") pod \"console-98f98995-rfsnl\" (UID: \"17586315-565c-4d7b-8f72-3bda38ccbe83\") " pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.077494 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.077450    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17586315-565c-4d7b-8f72-3bda38ccbe83-oauth-serving-cert\") pod \"console-98f98995-rfsnl\" (UID: \"17586315-565c-4d7b-8f72-3bda38ccbe83\") " pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.077695 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.077517    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcf7d\" (UniqueName: \"kubernetes.io/projected/17586315-565c-4d7b-8f72-3bda38ccbe83-kube-api-access-pcf7d\") pod \"console-98f98995-rfsnl\" (UID: \"17586315-565c-4d7b-8f72-3bda38ccbe83\") " pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.178543 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.178457    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17586315-565c-4d7b-8f72-3bda38ccbe83-console-oauth-config\") pod \"console-98f98995-rfsnl\" (UID: \"17586315-565c-4d7b-8f72-3bda38ccbe83\") " pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.178543 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.178499    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17586315-565c-4d7b-8f72-3bda38ccbe83-trusted-ca-bundle\") pod \"console-98f98995-rfsnl\" (UID: \"17586315-565c-4d7b-8f72-3bda38ccbe83\") " pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.178543 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.178518    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17586315-565c-4d7b-8f72-3bda38ccbe83-console-serving-cert\") pod \"console-98f98995-rfsnl\" (UID: \"17586315-565c-4d7b-8f72-3bda38ccbe83\") " pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.178543 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.178537    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17586315-565c-4d7b-8f72-3bda38ccbe83-oauth-serving-cert\") pod \"console-98f98995-rfsnl\" (UID: \"17586315-565c-4d7b-8f72-3bda38ccbe83\") " pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.178872 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.178694    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcf7d\" (UniqueName: \"kubernetes.io/projected/17586315-565c-4d7b-8f72-3bda38ccbe83-kube-api-access-pcf7d\") pod \"console-98f98995-rfsnl\" (UID: \"17586315-565c-4d7b-8f72-3bda38ccbe83\") " pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.178872 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.178781    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17586315-565c-4d7b-8f72-3bda38ccbe83-tmp\") pod \"console-98f98995-rfsnl\" (UID: \"17586315-565c-4d7b-8f72-3bda38ccbe83\") " pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.178872 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.178836    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17586315-565c-4d7b-8f72-3bda38ccbe83-service-ca\") pod \"console-98f98995-rfsnl\" (UID: \"17586315-565c-4d7b-8f72-3bda38ccbe83\") " pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.178872 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.178863    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17586315-565c-4d7b-8f72-3bda38ccbe83-console-config\") pod \"console-98f98995-rfsnl\" (UID: \"17586315-565c-4d7b-8f72-3bda38ccbe83\") " pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.179399 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.179374    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17586315-565c-4d7b-8f72-3bda38ccbe83-oauth-serving-cert\") pod \"console-98f98995-rfsnl\" (UID: \"17586315-565c-4d7b-8f72-3bda38ccbe83\") " pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.179606 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.179585    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17586315-565c-4d7b-8f72-3bda38ccbe83-trusted-ca-bundle\") pod \"console-98f98995-rfsnl\" (UID: \"17586315-565c-4d7b-8f72-3bda38ccbe83\") " pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.179680 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.179639    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17586315-565c-4d7b-8f72-3bda38ccbe83-console-config\") pod \"console-98f98995-rfsnl\" (UID: \"17586315-565c-4d7b-8f72-3bda38ccbe83\") " pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.179770 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.179752    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17586315-565c-4d7b-8f72-3bda38ccbe83-tmp\") pod \"console-98f98995-rfsnl\" (UID: \"17586315-565c-4d7b-8f72-3bda38ccbe83\") " pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.179804 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.179758    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17586315-565c-4d7b-8f72-3bda38ccbe83-service-ca\") pod \"console-98f98995-rfsnl\" (UID: \"17586315-565c-4d7b-8f72-3bda38ccbe83\") " pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.181220 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.181200    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17586315-565c-4d7b-8f72-3bda38ccbe83-console-serving-cert\") pod \"console-98f98995-rfsnl\" (UID: \"17586315-565c-4d7b-8f72-3bda38ccbe83\") " pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.181301 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.181203    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17586315-565c-4d7b-8f72-3bda38ccbe83-console-oauth-config\") pod \"console-98f98995-rfsnl\" (UID: \"17586315-565c-4d7b-8f72-3bda38ccbe83\") " pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.240408 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.240375    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcf7d\" (UniqueName: \"kubernetes.io/projected/17586315-565c-4d7b-8f72-3bda38ccbe83-kube-api-access-pcf7d\") pod \"console-98f98995-rfsnl\" (UID: \"17586315-565c-4d7b-8f72-3bda38ccbe83\") " pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.273406 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.273374    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:48.665104 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:48.665026    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-98f98995-rfsnl"]
Oct 31 03:20:48.667674 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:20:48.667641    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17586315_565c_4d7b_8f72_3bda38ccbe83.slice/crio-f1330ec08d67002a82536a21992f42ec35bbcaee1f078cc6dc9aa60b380357ad WatchSource:0}: Error finding container f1330ec08d67002a82536a21992f42ec35bbcaee1f078cc6dc9aa60b380357ad: Status 404 returned error can't find the container with id f1330ec08d67002a82536a21992f42ec35bbcaee1f078cc6dc9aa60b380357ad
Oct 31 03:20:49.001857 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:20:49.001827    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:20:49.005546 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:49.005514    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-98f98995-rfsnl" event={"ID":"17586315-565c-4d7b-8f72-3bda38ccbe83","Type":"ContainerStarted","Data":"807cc2be2301430b337bcf048bd2bcd6727060c5a63b5b38c25908eecd81bfbb"}
Oct 31 03:20:49.005717 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:49.005564    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-98f98995-rfsnl" event={"ID":"17586315-565c-4d7b-8f72-3bda38ccbe83","Type":"ContainerStarted","Data":"f1330ec08d67002a82536a21992f42ec35bbcaee1f078cc6dc9aa60b380357ad"}
Oct 31 03:20:49.125053 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:49.125007    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-98f98995-rfsnl" podStartSLOduration=2.124993116 podStartE2EDuration="2.124993116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:49.123729201 +0000 UTC m=+74.634711578" watchObservedRunningTime="2026-10-31 03:20:49.124993116 +0000 UTC m=+74.635975491"
Oct 31 03:20:52.617113 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:52.617072    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 03:20:52.617500 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:52.617449    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="alertmanager" containerID="cri-o://e1034fa2c52e06903051a4b34ad0de7873bcfd4d984e7a89f7d28ef9ae3ff24f" gracePeriod=120
Oct 31 03:20:52.617568 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:52.617505    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="kube-rbac-proxy-metric" containerID="cri-o://4a096211e692a0e95f9e56f42920fb738be670ece24bcb9336a352aaf68bcc4b" gracePeriod=120
Oct 31 03:20:52.617622 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:52.617542    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="config-reloader" containerID="cri-o://38da9ef6df6f0a3be404101c143951654ffb8078e6f2f2cbe01e92c4ae037a75" gracePeriod=120
Oct 31 03:20:52.617622 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:52.617558    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="kube-rbac-proxy" containerID="cri-o://1ee61aa324292d914d10b5cf9540aa3b241e47c439c714606796bb5f6138aa6e" gracePeriod=120
Oct 31 03:20:52.617622 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:52.617575    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="prom-label-proxy" containerID="cri-o://75066256a247ed923a17f85af18049d413100cb4f3def0dd70c863b8290e4616" gracePeriod=120
Oct 31 03:20:52.617752 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:52.617515    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="kube-rbac-proxy-web" containerID="cri-o://a2064bc248b06923e538581bf7d24685d8eeb07727d2ff69dda424d420fc8383" gracePeriod=120
Oct 31 03:20:53.023603 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:53.023569    4447 generic.go:356] "Generic (PLEG): container finished" podID="6b20027a-3439-4f84-8407-a9732ee5d748" containerID="75066256a247ed923a17f85af18049d413100cb4f3def0dd70c863b8290e4616" exitCode=0
Oct 31 03:20:53.023603 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:53.023593    4447 generic.go:356] "Generic (PLEG): container finished" podID="6b20027a-3439-4f84-8407-a9732ee5d748" containerID="4a096211e692a0e95f9e56f42920fb738be670ece24bcb9336a352aaf68bcc4b" exitCode=0
Oct 31 03:20:53.023603 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:53.023599    4447 generic.go:356] "Generic (PLEG): container finished" podID="6b20027a-3439-4f84-8407-a9732ee5d748" containerID="1ee61aa324292d914d10b5cf9540aa3b241e47c439c714606796bb5f6138aa6e" exitCode=0
Oct 31 03:20:53.023603 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:53.023604    4447 generic.go:356] "Generic (PLEG): container finished" podID="6b20027a-3439-4f84-8407-a9732ee5d748" containerID="38da9ef6df6f0a3be404101c143951654ffb8078e6f2f2cbe01e92c4ae037a75" exitCode=0
Oct 31 03:20:53.023603 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:53.023609    4447 generic.go:356] "Generic (PLEG): container finished" podID="6b20027a-3439-4f84-8407-a9732ee5d748" containerID="e1034fa2c52e06903051a4b34ad0de7873bcfd4d984e7a89f7d28ef9ae3ff24f" exitCode=0
Oct 31 03:20:53.023875 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:53.023640    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b20027a-3439-4f84-8407-a9732ee5d748","Type":"ContainerDied","Data":"75066256a247ed923a17f85af18049d413100cb4f3def0dd70c863b8290e4616"}
Oct 31 03:20:53.023875 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:53.023699    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b20027a-3439-4f84-8407-a9732ee5d748","Type":"ContainerDied","Data":"4a096211e692a0e95f9e56f42920fb738be670ece24bcb9336a352aaf68bcc4b"}
Oct 31 03:20:53.023875 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:53.023724    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b20027a-3439-4f84-8407-a9732ee5d748","Type":"ContainerDied","Data":"1ee61aa324292d914d10b5cf9540aa3b241e47c439c714606796bb5f6138aa6e"}
Oct 31 03:20:53.023875 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:53.023747    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b20027a-3439-4f84-8407-a9732ee5d748","Type":"ContainerDied","Data":"38da9ef6df6f0a3be404101c143951654ffb8078e6f2f2cbe01e92c4ae037a75"}
Oct 31 03:20:53.023875 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:53.023768    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b20027a-3439-4f84-8407-a9732ee5d748","Type":"ContainerDied","Data":"e1034fa2c52e06903051a4b34ad0de7873bcfd4d984e7a89f7d28ef9ae3ff24f"}
Oct 31 03:20:53.883370 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:53.883343    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.030143 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.030102    4447 generic.go:356] "Generic (PLEG): container finished" podID="6b20027a-3439-4f84-8407-a9732ee5d748" containerID="a2064bc248b06923e538581bf7d24685d8eeb07727d2ff69dda424d420fc8383" exitCode=0
Oct 31 03:20:54.030322 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.030167    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b20027a-3439-4f84-8407-a9732ee5d748","Type":"ContainerDied","Data":"a2064bc248b06923e538581bf7d24685d8eeb07727d2ff69dda424d420fc8383"}
Oct 31 03:20:54.030322 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.030184    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.030322 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.030239    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b20027a-3439-4f84-8407-a9732ee5d748","Type":"ContainerDied","Data":"65153e9e52c3dab58e21a11672e9d054b6fb5e80506a81be37b4933d1e9ef33b"}
Oct 31 03:20:54.030322 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.030273    4447 scope.go:122] "RemoveContainer" containerID="75066256a247ed923a17f85af18049d413100cb4f3def0dd70c863b8290e4616"
Oct 31 03:20:54.032338 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.032323    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-kube-rbac-proxy-metric\") pod \"6b20027a-3439-4f84-8407-a9732ee5d748\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") "
Oct 31 03:20:54.032424 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.032350    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b20027a-3439-4f84-8407-a9732ee5d748-kube-api-access-txzbc\" (UniqueName: \"kubernetes.io/projected/6b20027a-3439-4f84-8407-a9732ee5d748-kube-api-access-txzbc\") pod \"6b20027a-3439-4f84-8407-a9732ee5d748\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") "
Oct 31 03:20:54.032424 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.032386    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-main-tls\") pod \"6b20027a-3439-4f84-8407-a9732ee5d748\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") "
Oct 31 03:20:54.032424 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.032411    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-kube-rbac-proxy-web\") pod \"6b20027a-3439-4f84-8407-a9732ee5d748\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") "
Oct 31 03:20:54.032558 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.032539    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b20027a-3439-4f84-8407-a9732ee5d748-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6b20027a-3439-4f84-8407-a9732ee5d748-alertmanager-main-db\") pod \"6b20027a-3439-4f84-8407-a9732ee5d748\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") "
Oct 31 03:20:54.032611 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.032585    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-kube-rbac-proxy\") pod \"6b20027a-3439-4f84-8407-a9732ee5d748\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") "
Oct 31 03:20:54.032663 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.032633    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b20027a-3439-4f84-8407-a9732ee5d748-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b20027a-3439-4f84-8407-a9732ee5d748-alertmanager-trusted-ca-bundle\") pod \"6b20027a-3439-4f84-8407-a9732ee5d748\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") "
Oct 31 03:20:54.032717 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.032669    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b20027a-3439-4f84-8407-a9732ee5d748-tls-assets\" (UniqueName: \"kubernetes.io/projected/6b20027a-3439-4f84-8407-a9732ee5d748-tls-assets\") pod \"6b20027a-3439-4f84-8407-a9732ee5d748\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") "
Oct 31 03:20:54.032771 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.032719    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-web-config\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-web-config\") pod \"6b20027a-3439-4f84-8407-a9732ee5d748\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") "
Oct 31 03:20:54.032771 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.032746    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-config-volume\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-config-volume\") pod \"6b20027a-3439-4f84-8407-a9732ee5d748\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") "
Oct 31 03:20:54.032862 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.032780    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b20027a-3439-4f84-8407-a9732ee5d748-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b20027a-3439-4f84-8407-a9732ee5d748-metrics-client-ca\") pod \"6b20027a-3439-4f84-8407-a9732ee5d748\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") "
Oct 31 03:20:54.032862 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.032826    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b20027a-3439-4f84-8407-a9732ee5d748-config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b20027a-3439-4f84-8407-a9732ee5d748-config-out\") pod \"6b20027a-3439-4f84-8407-a9732ee5d748\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") "
Oct 31 03:20:54.032862 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.032845    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b20027a-3439-4f84-8407-a9732ee5d748-alertmanager-main-db" pod "6b20027a-3439-4f84-8407-a9732ee5d748" (UID: "6b20027a-3439-4f84-8407-a9732ee5d748"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:20:54.033022 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.032902    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-cluster-tls-config\") pod \"6b20027a-3439-4f84-8407-a9732ee5d748\" (UID: \"6b20027a-3439-4f84-8407-a9732ee5d748\") "
Oct 31 03:20:54.033391 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.033175    4447 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6b20027a-3439-4f84-8407-a9732ee5d748-alertmanager-main-db\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:20:54.033704 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.033674    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b20027a-3439-4f84-8407-a9732ee5d748-alertmanager-trusted-ca-bundle" pod "6b20027a-3439-4f84-8407-a9732ee5d748" (UID: "6b20027a-3439-4f84-8407-a9732ee5d748"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:20:54.033906 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.033860    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b20027a-3439-4f84-8407-a9732ee5d748-metrics-client-ca" pod "6b20027a-3439-4f84-8407-a9732ee5d748" (UID: "6b20027a-3439-4f84-8407-a9732ee5d748"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:20:54.035184 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.035160    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-main-tls" pod "6b20027a-3439-4f84-8407-a9732ee5d748" (UID: "6b20027a-3439-4f84-8407-a9732ee5d748"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:20:54.035356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.035327    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-kube-rbac-proxy-web" pod "6b20027a-3439-4f84-8407-a9732ee5d748" (UID: "6b20027a-3439-4f84-8407-a9732ee5d748"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:20:54.035435 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.035404    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b20027a-3439-4f84-8407-a9732ee5d748-kube-api-access-txzbc" pod "6b20027a-3439-4f84-8407-a9732ee5d748" (UID: "6b20027a-3439-4f84-8407-a9732ee5d748"). InnerVolumeSpecName "kube-api-access-txzbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:20:54.036165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.036139    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-kube-rbac-proxy-metric" pod "6b20027a-3439-4f84-8407-a9732ee5d748" (UID: "6b20027a-3439-4f84-8407-a9732ee5d748"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:20:54.036790 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.036764    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b20027a-3439-4f84-8407-a9732ee5d748-config-out" pod "6b20027a-3439-4f84-8407-a9732ee5d748" (UID: "6b20027a-3439-4f84-8407-a9732ee5d748"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:20:54.036964 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.036839    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-config-volume" pod "6b20027a-3439-4f84-8407-a9732ee5d748" (UID: "6b20027a-3439-4f84-8407-a9732ee5d748"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:20:54.037214 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.037192    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-kube-rbac-proxy" pod "6b20027a-3439-4f84-8407-a9732ee5d748" (UID: "6b20027a-3439-4f84-8407-a9732ee5d748"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:20:54.038507 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.038475    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b20027a-3439-4f84-8407-a9732ee5d748-tls-assets" pod "6b20027a-3439-4f84-8407-a9732ee5d748" (UID: "6b20027a-3439-4f84-8407-a9732ee5d748"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:20:54.041261 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.041085    4447 scope.go:122] "RemoveContainer" containerID="4a096211e692a0e95f9e56f42920fb738be670ece24bcb9336a352aaf68bcc4b"
Oct 31 03:20:54.044295 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.044267    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-cluster-tls-config" pod "6b20027a-3439-4f84-8407-a9732ee5d748" (UID: "6b20027a-3439-4f84-8407-a9732ee5d748"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:20:54.048825 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.048800    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-web-config" pod "6b20027a-3439-4f84-8407-a9732ee5d748" (UID: "6b20027a-3439-4f84-8407-a9732ee5d748"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:20:54.055585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.055567    4447 scope.go:122] "RemoveContainer" containerID="1ee61aa324292d914d10b5cf9540aa3b241e47c439c714606796bb5f6138aa6e"
Oct 31 03:20:54.062385 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.062363    4447 scope.go:122] "RemoveContainer" containerID="a2064bc248b06923e538581bf7d24685d8eeb07727d2ff69dda424d420fc8383"
Oct 31 03:20:54.069122 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.069103    4447 scope.go:122] "RemoveContainer" containerID="38da9ef6df6f0a3be404101c143951654ffb8078e6f2f2cbe01e92c4ae037a75"
Oct 31 03:20:54.076659 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.076642    4447 scope.go:122] "RemoveContainer" containerID="e1034fa2c52e06903051a4b34ad0de7873bcfd4d984e7a89f7d28ef9ae3ff24f"
Oct 31 03:20:54.083976 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.083959    4447 scope.go:122] "RemoveContainer" containerID="8f5ebe275607e8aa7163bc912eeb1f7f24072d3014eb67faf25e5e28d90263e0"
Oct 31 03:20:54.090980 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.090961    4447 scope.go:122] "RemoveContainer" containerID="75066256a247ed923a17f85af18049d413100cb4f3def0dd70c863b8290e4616"
Oct 31 03:20:54.091295 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:20:54.091274    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75066256a247ed923a17f85af18049d413100cb4f3def0dd70c863b8290e4616\": container with ID starting with 75066256a247ed923a17f85af18049d413100cb4f3def0dd70c863b8290e4616 not found: ID does not exist" containerID="75066256a247ed923a17f85af18049d413100cb4f3def0dd70c863b8290e4616"
Oct 31 03:20:54.091380 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.091304    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75066256a247ed923a17f85af18049d413100cb4f3def0dd70c863b8290e4616"} err="failed to get container status \"75066256a247ed923a17f85af18049d413100cb4f3def0dd70c863b8290e4616\": rpc error: code = NotFound desc = could not find container \"75066256a247ed923a17f85af18049d413100cb4f3def0dd70c863b8290e4616\": container with ID starting with 75066256a247ed923a17f85af18049d413100cb4f3def0dd70c863b8290e4616 not found: ID does not exist"
Oct 31 03:20:54.091380 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.091342    4447 scope.go:122] "RemoveContainer" containerID="4a096211e692a0e95f9e56f42920fb738be670ece24bcb9336a352aaf68bcc4b"
Oct 31 03:20:54.091619 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:20:54.091599    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a096211e692a0e95f9e56f42920fb738be670ece24bcb9336a352aaf68bcc4b\": container with ID starting with 4a096211e692a0e95f9e56f42920fb738be670ece24bcb9336a352aaf68bcc4b not found: ID does not exist" containerID="4a096211e692a0e95f9e56f42920fb738be670ece24bcb9336a352aaf68bcc4b"
Oct 31 03:20:54.091668 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.091626    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a096211e692a0e95f9e56f42920fb738be670ece24bcb9336a352aaf68bcc4b"} err="failed to get container status \"4a096211e692a0e95f9e56f42920fb738be670ece24bcb9336a352aaf68bcc4b\": rpc error: code = NotFound desc = could not find container \"4a096211e692a0e95f9e56f42920fb738be670ece24bcb9336a352aaf68bcc4b\": container with ID starting with 4a096211e692a0e95f9e56f42920fb738be670ece24bcb9336a352aaf68bcc4b not found: ID does not exist"
Oct 31 03:20:54.091668 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.091661    4447 scope.go:122] "RemoveContainer" containerID="1ee61aa324292d914d10b5cf9540aa3b241e47c439c714606796bb5f6138aa6e"
Oct 31 03:20:54.091914 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:20:54.091896    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ee61aa324292d914d10b5cf9540aa3b241e47c439c714606796bb5f6138aa6e\": container with ID starting with 1ee61aa324292d914d10b5cf9540aa3b241e47c439c714606796bb5f6138aa6e not found: ID does not exist" containerID="1ee61aa324292d914d10b5cf9540aa3b241e47c439c714606796bb5f6138aa6e"
Oct 31 03:20:54.091967 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.091921    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ee61aa324292d914d10b5cf9540aa3b241e47c439c714606796bb5f6138aa6e"} err="failed to get container status \"1ee61aa324292d914d10b5cf9540aa3b241e47c439c714606796bb5f6138aa6e\": rpc error: code = NotFound desc = could not find container \"1ee61aa324292d914d10b5cf9540aa3b241e47c439c714606796bb5f6138aa6e\": container with ID starting with 1ee61aa324292d914d10b5cf9540aa3b241e47c439c714606796bb5f6138aa6e not found: ID does not exist"
Oct 31 03:20:54.091967 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.091939    4447 scope.go:122] "RemoveContainer" containerID="a2064bc248b06923e538581bf7d24685d8eeb07727d2ff69dda424d420fc8383"
Oct 31 03:20:54.092169 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:20:54.092153    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2064bc248b06923e538581bf7d24685d8eeb07727d2ff69dda424d420fc8383\": container with ID starting with a2064bc248b06923e538581bf7d24685d8eeb07727d2ff69dda424d420fc8383 not found: ID does not exist" containerID="a2064bc248b06923e538581bf7d24685d8eeb07727d2ff69dda424d420fc8383"
Oct 31 03:20:54.092209 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.092172    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2064bc248b06923e538581bf7d24685d8eeb07727d2ff69dda424d420fc8383"} err="failed to get container status \"a2064bc248b06923e538581bf7d24685d8eeb07727d2ff69dda424d420fc8383\": rpc error: code = NotFound desc = could not find container \"a2064bc248b06923e538581bf7d24685d8eeb07727d2ff69dda424d420fc8383\": container with ID starting with a2064bc248b06923e538581bf7d24685d8eeb07727d2ff69dda424d420fc8383 not found: ID does not exist"
Oct 31 03:20:54.092209 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.092186    4447 scope.go:122] "RemoveContainer" containerID="38da9ef6df6f0a3be404101c143951654ffb8078e6f2f2cbe01e92c4ae037a75"
Oct 31 03:20:54.092374 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:20:54.092359    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38da9ef6df6f0a3be404101c143951654ffb8078e6f2f2cbe01e92c4ae037a75\": container with ID starting with 38da9ef6df6f0a3be404101c143951654ffb8078e6f2f2cbe01e92c4ae037a75 not found: ID does not exist" containerID="38da9ef6df6f0a3be404101c143951654ffb8078e6f2f2cbe01e92c4ae037a75"
Oct 31 03:20:54.092410 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.092378    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38da9ef6df6f0a3be404101c143951654ffb8078e6f2f2cbe01e92c4ae037a75"} err="failed to get container status \"38da9ef6df6f0a3be404101c143951654ffb8078e6f2f2cbe01e92c4ae037a75\": rpc error: code = NotFound desc = could not find container \"38da9ef6df6f0a3be404101c143951654ffb8078e6f2f2cbe01e92c4ae037a75\": container with ID starting with 38da9ef6df6f0a3be404101c143951654ffb8078e6f2f2cbe01e92c4ae037a75 not found: ID does not exist"
Oct 31 03:20:54.092410 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.092390    4447 scope.go:122] "RemoveContainer" containerID="e1034fa2c52e06903051a4b34ad0de7873bcfd4d984e7a89f7d28ef9ae3ff24f"
Oct 31 03:20:54.092561 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:20:54.092545    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1034fa2c52e06903051a4b34ad0de7873bcfd4d984e7a89f7d28ef9ae3ff24f\": container with ID starting with e1034fa2c52e06903051a4b34ad0de7873bcfd4d984e7a89f7d28ef9ae3ff24f not found: ID does not exist" containerID="e1034fa2c52e06903051a4b34ad0de7873bcfd4d984e7a89f7d28ef9ae3ff24f"
Oct 31 03:20:54.092635 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.092565    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1034fa2c52e06903051a4b34ad0de7873bcfd4d984e7a89f7d28ef9ae3ff24f"} err="failed to get container status \"e1034fa2c52e06903051a4b34ad0de7873bcfd4d984e7a89f7d28ef9ae3ff24f\": rpc error: code = NotFound desc = could not find container \"e1034fa2c52e06903051a4b34ad0de7873bcfd4d984e7a89f7d28ef9ae3ff24f\": container with ID starting with e1034fa2c52e06903051a4b34ad0de7873bcfd4d984e7a89f7d28ef9ae3ff24f not found: ID does not exist"
Oct 31 03:20:54.092635 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.092576    4447 scope.go:122] "RemoveContainer" containerID="8f5ebe275607e8aa7163bc912eeb1f7f24072d3014eb67faf25e5e28d90263e0"
Oct 31 03:20:54.092759 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:20:54.092739    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f5ebe275607e8aa7163bc912eeb1f7f24072d3014eb67faf25e5e28d90263e0\": container with ID starting with 8f5ebe275607e8aa7163bc912eeb1f7f24072d3014eb67faf25e5e28d90263e0 not found: ID does not exist" containerID="8f5ebe275607e8aa7163bc912eeb1f7f24072d3014eb67faf25e5e28d90263e0"
Oct 31 03:20:54.092795 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.092764    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f5ebe275607e8aa7163bc912eeb1f7f24072d3014eb67faf25e5e28d90263e0"} err="failed to get container status \"8f5ebe275607e8aa7163bc912eeb1f7f24072d3014eb67faf25e5e28d90263e0\": rpc error: code = NotFound desc = could not find container \"8f5ebe275607e8aa7163bc912eeb1f7f24072d3014eb67faf25e5e28d90263e0\": container with ID starting with 8f5ebe275607e8aa7163bc912eeb1f7f24072d3014eb67faf25e5e28d90263e0 not found: ID does not exist"
Oct 31 03:20:54.134925 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.134876    4447 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:20:54.134925 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.134923    4447 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:20:54.135116 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.134937    4447 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b20027a-3439-4f84-8407-a9732ee5d748-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:20:54.135116 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.134947    4447 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b20027a-3439-4f84-8407-a9732ee5d748-tls-assets\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:20:54.135116 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.134956    4447 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-web-config\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:20:54.135116 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.134963    4447 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:20:54.135116 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.134971    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b20027a-3439-4f84-8407-a9732ee5d748-metrics-client-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:20:54.135116 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.134979    4447 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b20027a-3439-4f84-8407-a9732ee5d748-config-out\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:20:54.135116 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.134987    4447 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-cluster-tls-config\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:20:54.135116 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.134999    4447 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:20:54.135116 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.135008    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txzbc\" (UniqueName: \"kubernetes.io/projected/6b20027a-3439-4f84-8407-a9732ee5d748-kube-api-access-txzbc\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:20:54.135116 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.135016    4447 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b20027a-3439-4f84-8407-a9732ee5d748-secret-alertmanager-main-tls\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:20:54.452562 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.452528    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 03:20:54.457326 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.457306    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 03:20:54.488658 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.488631    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 03:20:54.488970 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.488957    4447 cpu_manager.go:414] "containerMap: removing container" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="config-reloader"
Oct 31 03:20:54.489026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.488972    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="config-reloader"
Oct 31 03:20:54.489026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.488986    4447 cpu_manager.go:414] "containerMap: removing container" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="kube-rbac-proxy-metric"
Oct 31 03:20:54.489026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.488990    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="kube-rbac-proxy-metric"
Oct 31 03:20:54.489026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.489004    4447 cpu_manager.go:414] "containerMap: removing container" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="kube-rbac-proxy-web"
Oct 31 03:20:54.489026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.489008    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="kube-rbac-proxy-web"
Oct 31 03:20:54.489026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.489014    4447 cpu_manager.go:414] "containerMap: removing container" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="alertmanager"
Oct 31 03:20:54.489026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.489018    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="alertmanager"
Oct 31 03:20:54.489026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.489025    4447 cpu_manager.go:414] "containerMap: removing container" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="prom-label-proxy"
Oct 31 03:20:54.489026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.489030    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="prom-label-proxy"
Oct 31 03:20:54.489297 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.489038    4447 cpu_manager.go:414] "containerMap: removing container" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="init-config-reloader"
Oct 31 03:20:54.489297 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.489043    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="init-config-reloader"
Oct 31 03:20:54.489297 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.489052    4447 cpu_manager.go:414] "containerMap: removing container" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="kube-rbac-proxy"
Oct 31 03:20:54.489297 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.489056    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="kube-rbac-proxy"
Oct 31 03:20:54.489297 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.489102    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="kube-rbac-proxy"
Oct 31 03:20:54.489297 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.489111    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="config-reloader"
Oct 31 03:20:54.489297 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.489118    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="kube-rbac-proxy-metric"
Oct 31 03:20:54.489297 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.489124    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="kube-rbac-proxy-web"
Oct 31 03:20:54.489297 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.489129    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="alertmanager"
Oct 31 03:20:54.489297 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.489134    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" containerName="prom-label-proxy"
Oct 31 03:20:54.524856 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.524823    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.577952 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.577910    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 03:20:54.577952 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.577943    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 03:20:54.578213 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.577945    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 03:20:54.578213 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.577976    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 03:20:54.578213 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.578082    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 03:20:54.578213 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.578081    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 03:20:54.578213 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.578106    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5bnqw\""
Oct 31 03:20:54.578213 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.578105    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 03:20:54.578213 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.578157    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 03:20:54.579151 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.579131    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 03:20:54.605004 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.604942    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b20027a-3439-4f84-8407-a9732ee5d748" path="/var/lib/kubelet/pods/6b20027a-3439-4f84-8407-a9732ee5d748/volumes"
Oct 31 03:20:54.628692 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.628663    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 03:20:54.638554 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.638529    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17f3e0fe-a1e5-44ec-a293-a557799c7da2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.638554 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.638557    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17f3e0fe-a1e5-44ec-a293-a557799c7da2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.638788 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.638596    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17f3e0fe-a1e5-44ec-a293-a557799c7da2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.638788 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.638670    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/17f3e0fe-a1e5-44ec-a293-a557799c7da2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.638788 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.638702    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dz6c\" (UniqueName: \"kubernetes.io/projected/17f3e0fe-a1e5-44ec-a293-a557799c7da2-kube-api-access-5dz6c\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.638977 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.638798    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/17f3e0fe-a1e5-44ec-a293-a557799c7da2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.638977 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.638829    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17f3e0fe-a1e5-44ec-a293-a557799c7da2-config-out\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.638977 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.638846    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17f3e0fe-a1e5-44ec-a293-a557799c7da2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.638977 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.638864    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17f3e0fe-a1e5-44ec-a293-a557799c7da2-web-config\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.638977 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.638908    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/17f3e0fe-a1e5-44ec-a293-a557799c7da2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.639225 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.639009    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/17f3e0fe-a1e5-44ec-a293-a557799c7da2-config-volume\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.639225 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.639060    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17f3e0fe-a1e5-44ec-a293-a557799c7da2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.639225 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.639091    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/17f3e0fe-a1e5-44ec-a293-a557799c7da2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.740207 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.740173    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17f3e0fe-a1e5-44ec-a293-a557799c7da2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.740207 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.740209    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/17f3e0fe-a1e5-44ec-a293-a557799c7da2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.740441 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.740224    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dz6c\" (UniqueName: \"kubernetes.io/projected/17f3e0fe-a1e5-44ec-a293-a557799c7da2-kube-api-access-5dz6c\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.740441 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.740272    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/17f3e0fe-a1e5-44ec-a293-a557799c7da2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.740441 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.740299    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17f3e0fe-a1e5-44ec-a293-a557799c7da2-config-out\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.740441 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.740323    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17f3e0fe-a1e5-44ec-a293-a557799c7da2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.740441 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.740349    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17f3e0fe-a1e5-44ec-a293-a557799c7da2-web-config\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.740441 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.740375    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/17f3e0fe-a1e5-44ec-a293-a557799c7da2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.740441 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.740419    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/17f3e0fe-a1e5-44ec-a293-a557799c7da2-config-volume\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.740441 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.740439    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17f3e0fe-a1e5-44ec-a293-a557799c7da2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.740826 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.740468    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/17f3e0fe-a1e5-44ec-a293-a557799c7da2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.740826 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.740543    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17f3e0fe-a1e5-44ec-a293-a557799c7da2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.740826 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.740578    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17f3e0fe-a1e5-44ec-a293-a557799c7da2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.740826 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.740651    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/17f3e0fe-a1e5-44ec-a293-a557799c7da2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.741083 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.741004    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17f3e0fe-a1e5-44ec-a293-a557799c7da2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.741385 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.741363    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17f3e0fe-a1e5-44ec-a293-a557799c7da2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.743822 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.743794    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17f3e0fe-a1e5-44ec-a293-a557799c7da2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.744005 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.743982    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17f3e0fe-a1e5-44ec-a293-a557799c7da2-web-config\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.744076 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.744023    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/17f3e0fe-a1e5-44ec-a293-a557799c7da2-config-volume\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.744132 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.744084    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/17f3e0fe-a1e5-44ec-a293-a557799c7da2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.744132 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.744097    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/17f3e0fe-a1e5-44ec-a293-a557799c7da2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.744234 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.744170    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17f3e0fe-a1e5-44ec-a293-a557799c7da2-config-out\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.744285 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.744233    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/17f3e0fe-a1e5-44ec-a293-a557799c7da2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.744443 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.744426    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17f3e0fe-a1e5-44ec-a293-a557799c7da2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.745707 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.745689    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17f3e0fe-a1e5-44ec-a293-a557799c7da2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.802670 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.802631    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dz6c\" (UniqueName: \"kubernetes.io/projected/17f3e0fe-a1e5-44ec-a293-a557799c7da2-kube-api-access-5dz6c\") pod \"alertmanager-main-0\" (UID: \"17f3e0fe-a1e5-44ec-a293-a557799c7da2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:54.972942 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:54.972914    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:20:55.486463 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:55.486441    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 03:20:55.488229 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:20:55.488199    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f3e0fe_a1e5_44ec_a293_a557799c7da2.slice/crio-b3804cfdda45fb2001f337230547c232ca5f2b8c80131239de6d9e2c364a22d0 WatchSource:0}: Error finding container b3804cfdda45fb2001f337230547c232ca5f2b8c80131239de6d9e2c364a22d0: Status 404 returned error can't find the container with id b3804cfdda45fb2001f337230547c232ca5f2b8c80131239de6d9e2c364a22d0
Oct 31 03:20:56.040200 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:56.040170    4447 generic.go:356] "Generic (PLEG): container finished" podID="17f3e0fe-a1e5-44ec-a293-a557799c7da2" containerID="5afa8981e51c554a5e19c5441372c809519f2b64038e93bbd2d3adf74be39bc2" exitCode=0
Oct 31 03:20:56.040590 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:56.040252    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17f3e0fe-a1e5-44ec-a293-a557799c7da2","Type":"ContainerDied","Data":"5afa8981e51c554a5e19c5441372c809519f2b64038e93bbd2d3adf74be39bc2"}
Oct 31 03:20:56.040590 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:56.040322    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17f3e0fe-a1e5-44ec-a293-a557799c7da2","Type":"ContainerStarted","Data":"b3804cfdda45fb2001f337230547c232ca5f2b8c80131239de6d9e2c364a22d0"}
Oct 31 03:20:57.046659 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:57.046622    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17f3e0fe-a1e5-44ec-a293-a557799c7da2","Type":"ContainerStarted","Data":"d575e3ada00737c2092bc4be798e2f4cf3fce6dbf1de3a20c7d0c9b0059870ac"}
Oct 31 03:20:57.047033 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:57.046701    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17f3e0fe-a1e5-44ec-a293-a557799c7da2","Type":"ContainerStarted","Data":"16c66f981e8b5ba39699441796c4a206fe140a38bb1f9bc86c40213a7248175a"}
Oct 31 03:20:57.047033 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:57.046739    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17f3e0fe-a1e5-44ec-a293-a557799c7da2","Type":"ContainerStarted","Data":"c0235f7c3f4366e127516bad8dfe14acb65dddf5f8cb480c5db48d8c352a6ee2"}
Oct 31 03:20:57.047033 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:57.046775    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17f3e0fe-a1e5-44ec-a293-a557799c7da2","Type":"ContainerStarted","Data":"7e6697daea163cc36a0dcb6a3134c212922e98c664265730333ce02b4f6550fa"}
Oct 31 03:20:57.047033 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:57.046813    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17f3e0fe-a1e5-44ec-a293-a557799c7da2","Type":"ContainerStarted","Data":"88390367e1088df828295be44b217b4ed66d3854c9beda649413f9063c41d4c8"}
Oct 31 03:20:57.047033 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:57.046846    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17f3e0fe-a1e5-44ec-a293-a557799c7da2","Type":"ContainerStarted","Data":"ada591f25253f904916ba41e1be4afe929adc47234c7b3004788faaf849f83dc"}
Oct 31 03:20:57.177973 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:57.177925    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.16617685 podStartE2EDuration="3.177912s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.73515ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:57.176632651 +0000 UTC m=+82.687615027" watchObservedRunningTime="2026-10-31 03:20:57.177912 +0000 UTC m=+82.688894356"
Oct 31 03:20:58.121853 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.121787    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7c656c8f7d-cprkw" podUID="7b7b96f3-2670-4021-a387-15f089d2f1d1" containerName="console" containerID="cri-o://f385bd1a3bf4f256d99c589caaf31dfac466d4112ed8fbe2f601f8840e10e2c3" gracePeriod=15
Oct 31 03:20:58.276804 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.275905    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:58.277020 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.276996    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:58.280439 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.280417    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:58.413067 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.413044    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:58.479706 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.479674    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b7b96f3-2670-4021-a387-15f089d2f1d1-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b7b96f3-2670-4021-a387-15f089d2f1d1-console-oauth-config\") pod \"7b7b96f3-2670-4021-a387-15f089d2f1d1\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") "
Oct 31 03:20:58.479852 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.479741    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b7b96f3-2670-4021-a387-15f089d2f1d1-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b7b96f3-2670-4021-a387-15f089d2f1d1-tmp\") pod \"7b7b96f3-2670-4021-a387-15f089d2f1d1\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") "
Oct 31 03:20:58.479852 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.479770    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b7b96f3-2670-4021-a387-15f089d2f1d1-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b7b96f3-2670-4021-a387-15f089d2f1d1-console-serving-cert\") pod \"7b7b96f3-2670-4021-a387-15f089d2f1d1\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") "
Oct 31 03:20:58.479852 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.479806    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b7b96f3-2670-4021-a387-15f089d2f1d1-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b7b96f3-2670-4021-a387-15f089d2f1d1-oauth-serving-cert\") pod \"7b7b96f3-2670-4021-a387-15f089d2f1d1\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") "
Oct 31 03:20:58.479852 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.479843    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b7b96f3-2670-4021-a387-15f089d2f1d1-console-config\" (UniqueName: \"kubernetes.io/configmap/7b7b96f3-2670-4021-a387-15f089d2f1d1-console-config\") pod \"7b7b96f3-2670-4021-a387-15f089d2f1d1\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") "
Oct 31 03:20:58.480106 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.479873    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b7b96f3-2670-4021-a387-15f089d2f1d1-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b7b96f3-2670-4021-a387-15f089d2f1d1-service-ca\") pod \"7b7b96f3-2670-4021-a387-15f089d2f1d1\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") "
Oct 31 03:20:58.480106 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.479974    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b7b96f3-2670-4021-a387-15f089d2f1d1-kube-api-access-zrdb2\" (UniqueName: \"kubernetes.io/projected/7b7b96f3-2670-4021-a387-15f089d2f1d1-kube-api-access-zrdb2\") pod \"7b7b96f3-2670-4021-a387-15f089d2f1d1\" (UID: \"7b7b96f3-2670-4021-a387-15f089d2f1d1\") "
Oct 31 03:20:58.480215 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.480103    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b7b96f3-2670-4021-a387-15f089d2f1d1-tmp" pod "7b7b96f3-2670-4021-a387-15f089d2f1d1" (UID: "7b7b96f3-2670-4021-a387-15f089d2f1d1"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:20:58.480346 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.480285    4447 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b7b96f3-2670-4021-a387-15f089d2f1d1-tmp\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:20:58.480346 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.480284    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b7b96f3-2670-4021-a387-15f089d2f1d1-oauth-serving-cert" pod "7b7b96f3-2670-4021-a387-15f089d2f1d1" (UID: "7b7b96f3-2670-4021-a387-15f089d2f1d1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:20:58.480346 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.480301    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b7b96f3-2670-4021-a387-15f089d2f1d1-console-config" pod "7b7b96f3-2670-4021-a387-15f089d2f1d1" (UID: "7b7b96f3-2670-4021-a387-15f089d2f1d1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:20:58.480346 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.480309    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b7b96f3-2670-4021-a387-15f089d2f1d1-service-ca" pod "7b7b96f3-2670-4021-a387-15f089d2f1d1" (UID: "7b7b96f3-2670-4021-a387-15f089d2f1d1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:20:58.482118 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.482097    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b7b96f3-2670-4021-a387-15f089d2f1d1-console-oauth-config" pod "7b7b96f3-2670-4021-a387-15f089d2f1d1" (UID: "7b7b96f3-2670-4021-a387-15f089d2f1d1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:20:58.482257 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.482234    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b7b96f3-2670-4021-a387-15f089d2f1d1-console-serving-cert" pod "7b7b96f3-2670-4021-a387-15f089d2f1d1" (UID: "7b7b96f3-2670-4021-a387-15f089d2f1d1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:20:58.482257 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.482249    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b7b96f3-2670-4021-a387-15f089d2f1d1-kube-api-access-zrdb2" pod "7b7b96f3-2670-4021-a387-15f089d2f1d1" (UID: "7b7b96f3-2670-4021-a387-15f089d2f1d1"). InnerVolumeSpecName "kube-api-access-zrdb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:20:58.581317 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.581284    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrdb2\" (UniqueName: \"kubernetes.io/projected/7b7b96f3-2670-4021-a387-15f089d2f1d1-kube-api-access-zrdb2\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:20:58.581317 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.581311    4447 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b7b96f3-2670-4021-a387-15f089d2f1d1-console-oauth-config\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:20:58.581317 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.581320    4447 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b7b96f3-2670-4021-a387-15f089d2f1d1-console-serving-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:20:58.581538 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.581329    4447 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b7b96f3-2670-4021-a387-15f089d2f1d1-oauth-serving-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:20:58.581538 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.581344    4447 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b7b96f3-2670-4021-a387-15f089d2f1d1-console-config\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:20:58.581538 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:58.581355    4447 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b7b96f3-2670-4021-a387-15f089d2f1d1-service-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:20:59.056575 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:59.056538    4447 generic.go:356] "Generic (PLEG): container finished" podID="7b7b96f3-2670-4021-a387-15f089d2f1d1" containerID="f385bd1a3bf4f256d99c589caaf31dfac466d4112ed8fbe2f601f8840e10e2c3" exitCode=0
Oct 31 03:20:59.056774 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:59.056592    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c656c8f7d-cprkw"
Oct 31 03:20:59.056774 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:59.056625    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c656c8f7d-cprkw" event={"ID":"7b7b96f3-2670-4021-a387-15f089d2f1d1","Type":"ContainerDied","Data":"f385bd1a3bf4f256d99c589caaf31dfac466d4112ed8fbe2f601f8840e10e2c3"}
Oct 31 03:20:59.056774 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:59.056689    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c656c8f7d-cprkw" event={"ID":"7b7b96f3-2670-4021-a387-15f089d2f1d1","Type":"ContainerDied","Data":"79f87b5cb777da675e6da0a320566c379c9bf67aeac22b5a997c27ad2a5d47da"}
Oct 31 03:20:59.056774 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:59.056714    4447 scope.go:122] "RemoveContainer" containerID="f385bd1a3bf4f256d99c589caaf31dfac466d4112ed8fbe2f601f8840e10e2c3"
Oct 31 03:20:59.060997 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:59.060972    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-98f98995-rfsnl"
Oct 31 03:20:59.065199 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:59.065180    4447 scope.go:122] "RemoveContainer" containerID="f385bd1a3bf4f256d99c589caaf31dfac466d4112ed8fbe2f601f8840e10e2c3"
Oct 31 03:20:59.065638 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:20:59.065540    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f385bd1a3bf4f256d99c589caaf31dfac466d4112ed8fbe2f601f8840e10e2c3\": container with ID starting with f385bd1a3bf4f256d99c589caaf31dfac466d4112ed8fbe2f601f8840e10e2c3 not found: ID does not exist" containerID="f385bd1a3bf4f256d99c589caaf31dfac466d4112ed8fbe2f601f8840e10e2c3"
Oct 31 03:20:59.065638 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:59.065578    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f385bd1a3bf4f256d99c589caaf31dfac466d4112ed8fbe2f601f8840e10e2c3"} err="failed to get container status \"f385bd1a3bf4f256d99c589caaf31dfac466d4112ed8fbe2f601f8840e10e2c3\": rpc error: code = NotFound desc = could not find container \"f385bd1a3bf4f256d99c589caaf31dfac466d4112ed8fbe2f601f8840e10e2c3\": container with ID starting with f385bd1a3bf4f256d99c589caaf31dfac466d4112ed8fbe2f601f8840e10e2c3 not found: ID does not exist"
Oct 31 03:20:59.221744 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:59.221705    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c656c8f7d-cprkw"]
Oct 31 03:20:59.271724 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:59.271692    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c656c8f7d-cprkw"]
Oct 31 03:20:59.423543 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:20:59.423460    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d99d6f6d8-n4w55"]
Oct 31 03:21:00.605511 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:00.605479    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b7b96f3-2670-4021-a387-15f089d2f1d1" path="/var/lib/kubelet/pods/7b7b96f3-2670-4021-a387-15f089d2f1d1/volumes"
Oct 31 03:21:19.001495 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:21:19.001457    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:21:24.447773 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.447712    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6d99d6f6d8-n4w55" podUID="64782f21-9364-4cd2-9731-cd27f2ba2c29" containerName="console" containerID="cri-o://b46aa23029c3551f5b3ed990bff2aef96a5f7291c4aff5b3ec8bea0c6cba96b0" gracePeriod=15
Oct 31 03:21:24.690673 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.690649    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:21:24.796054 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.796021    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64782f21-9364-4cd2-9731-cd27f2ba2c29-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64782f21-9364-4cd2-9731-cd27f2ba2c29-console-serving-cert\") pod \"64782f21-9364-4cd2-9731-cd27f2ba2c29\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") "
Oct 31 03:21:24.796230 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.796089    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-console-config\" (UniqueName: \"kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-console-config\") pod \"64782f21-9364-4cd2-9731-cd27f2ba2c29\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") "
Oct 31 03:21:24.796230 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.796116    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64782f21-9364-4cd2-9731-cd27f2ba2c29-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64782f21-9364-4cd2-9731-cd27f2ba2c29-console-oauth-config\") pod \"64782f21-9364-4cd2-9731-cd27f2ba2c29\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") "
Oct 31 03:21:24.796335 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.796293    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-oauth-serving-cert\") pod \"64782f21-9364-4cd2-9731-cd27f2ba2c29\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") "
Oct 31 03:21:24.796394 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.796355    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64782f21-9364-4cd2-9731-cd27f2ba2c29-kube-api-access-r5tcb\" (UniqueName: \"kubernetes.io/projected/64782f21-9364-4cd2-9731-cd27f2ba2c29-kube-api-access-r5tcb\") pod \"64782f21-9364-4cd2-9731-cd27f2ba2c29\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") "
Oct 31 03:21:24.796450 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.796390    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64782f21-9364-4cd2-9731-cd27f2ba2c29-tmp\" (UniqueName: \"kubernetes.io/empty-dir/64782f21-9364-4cd2-9731-cd27f2ba2c29-tmp\") pod \"64782f21-9364-4cd2-9731-cd27f2ba2c29\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") "
Oct 31 03:21:24.796450 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.796431    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-service-ca\" (UniqueName: \"kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-service-ca\") pod \"64782f21-9364-4cd2-9731-cd27f2ba2c29\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") "
Oct 31 03:21:24.796547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.796462    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-trusted-ca-bundle\") pod \"64782f21-9364-4cd2-9731-cd27f2ba2c29\" (UID: \"64782f21-9364-4cd2-9731-cd27f2ba2c29\") "
Oct 31 03:21:24.796547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.796528    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-console-config" pod "64782f21-9364-4cd2-9731-cd27f2ba2c29" (UID: "64782f21-9364-4cd2-9731-cd27f2ba2c29"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:21:24.796662 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.796638    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-oauth-serving-cert" pod "64782f21-9364-4cd2-9731-cd27f2ba2c29" (UID: "64782f21-9364-4cd2-9731-cd27f2ba2c29"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:21:24.796831 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.796789    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64782f21-9364-4cd2-9731-cd27f2ba2c29-tmp" pod "64782f21-9364-4cd2-9731-cd27f2ba2c29" (UID: "64782f21-9364-4cd2-9731-cd27f2ba2c29"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:21:24.797017 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.796989    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-trusted-ca-bundle" pod "64782f21-9364-4cd2-9731-cd27f2ba2c29" (UID: "64782f21-9364-4cd2-9731-cd27f2ba2c29"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:21:24.797100 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.797018    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-service-ca" pod "64782f21-9364-4cd2-9731-cd27f2ba2c29" (UID: "64782f21-9364-4cd2-9731-cd27f2ba2c29"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:21:24.797100 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.797034    4447 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-console-config\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:21:24.797100 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.797057    4447 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-oauth-serving-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:21:24.798383 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.798364    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64782f21-9364-4cd2-9731-cd27f2ba2c29-console-oauth-config" pod "64782f21-9364-4cd2-9731-cd27f2ba2c29" (UID: "64782f21-9364-4cd2-9731-cd27f2ba2c29"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:21:24.798912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.798875    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64782f21-9364-4cd2-9731-cd27f2ba2c29-kube-api-access-r5tcb" pod "64782f21-9364-4cd2-9731-cd27f2ba2c29" (UID: "64782f21-9364-4cd2-9731-cd27f2ba2c29"). InnerVolumeSpecName "kube-api-access-r5tcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:21:24.798912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.798874    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64782f21-9364-4cd2-9731-cd27f2ba2c29-console-serving-cert" pod "64782f21-9364-4cd2-9731-cd27f2ba2c29" (UID: "64782f21-9364-4cd2-9731-cd27f2ba2c29"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:21:24.898375 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.898336    4447 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64782f21-9364-4cd2-9731-cd27f2ba2c29-console-oauth-config\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:21:24.898375 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.898365    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5tcb\" (UniqueName: \"kubernetes.io/projected/64782f21-9364-4cd2-9731-cd27f2ba2c29-kube-api-access-r5tcb\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:21:24.898375 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.898375    4447 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64782f21-9364-4cd2-9731-cd27f2ba2c29-tmp\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:21:24.898375 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.898385    4447 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-service-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:21:24.898631 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.898393    4447 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64782f21-9364-4cd2-9731-cd27f2ba2c29-trusted-ca-bundle\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:21:24.898631 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:24.898401    4447 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64782f21-9364-4cd2-9731-cd27f2ba2c29-console-serving-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:21:25.152983 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:25.152870    4447 generic.go:356] "Generic (PLEG): container finished" podID="64782f21-9364-4cd2-9731-cd27f2ba2c29" containerID="b46aa23029c3551f5b3ed990bff2aef96a5f7291c4aff5b3ec8bea0c6cba96b0" exitCode=0
Oct 31 03:21:25.152983 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:25.152950    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d99d6f6d8-n4w55"
Oct 31 03:21:25.152983 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:25.152951    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d99d6f6d8-n4w55" event={"ID":"64782f21-9364-4cd2-9731-cd27f2ba2c29","Type":"ContainerDied","Data":"b46aa23029c3551f5b3ed990bff2aef96a5f7291c4aff5b3ec8bea0c6cba96b0"}
Oct 31 03:21:25.153219 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:25.153037    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d99d6f6d8-n4w55" event={"ID":"64782f21-9364-4cd2-9731-cd27f2ba2c29","Type":"ContainerDied","Data":"59f1aafad753210a9cf82e490eace13345db812ebd0e16fae786b22a4813fe12"}
Oct 31 03:21:25.153219 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:25.153060    4447 scope.go:122] "RemoveContainer" containerID="b46aa23029c3551f5b3ed990bff2aef96a5f7291c4aff5b3ec8bea0c6cba96b0"
Oct 31 03:21:25.163782 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:25.163766    4447 scope.go:122] "RemoveContainer" containerID="b46aa23029c3551f5b3ed990bff2aef96a5f7291c4aff5b3ec8bea0c6cba96b0"
Oct 31 03:21:25.164062 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:21:25.164042    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b46aa23029c3551f5b3ed990bff2aef96a5f7291c4aff5b3ec8bea0c6cba96b0\": container with ID starting with b46aa23029c3551f5b3ed990bff2aef96a5f7291c4aff5b3ec8bea0c6cba96b0 not found: ID does not exist" containerID="b46aa23029c3551f5b3ed990bff2aef96a5f7291c4aff5b3ec8bea0c6cba96b0"
Oct 31 03:21:25.164143 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:25.164076    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b46aa23029c3551f5b3ed990bff2aef96a5f7291c4aff5b3ec8bea0c6cba96b0"} err="failed to get container status \"b46aa23029c3551f5b3ed990bff2aef96a5f7291c4aff5b3ec8bea0c6cba96b0\": rpc error: code = NotFound desc = could not find container \"b46aa23029c3551f5b3ed990bff2aef96a5f7291c4aff5b3ec8bea0c6cba96b0\": container with ID starting with b46aa23029c3551f5b3ed990bff2aef96a5f7291c4aff5b3ec8bea0c6cba96b0 not found: ID does not exist"
Oct 31 03:21:25.318138 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:25.318101    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d99d6f6d8-n4w55"]
Oct 31 03:21:25.368023 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:25.367991    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d99d6f6d8-n4w55"]
Oct 31 03:21:26.605226 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:21:26.605196    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64782f21-9364-4cd2-9731-cd27f2ba2c29" path="/var/lib/kubelet/pods/64782f21-9364-4cd2-9731-cd27f2ba2c29/volumes"
Oct 31 03:21:49.000815 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:21:49.000732    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:22:19.001397 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:22:19.001358    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:22:49.001066 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:22:49.001030    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:23:19.001872 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:23:19.001791    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:23:19.339988 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:19.339914    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql"]
Oct 31 03:23:19.340251 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:19.340240    4447 cpu_manager.go:414] "containerMap: removing container" podUID="7b7b96f3-2670-4021-a387-15f089d2f1d1" containerName="console"
Oct 31 03:23:19.340291 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:19.340253    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b7b96f3-2670-4021-a387-15f089d2f1d1" containerName="console"
Oct 31 03:23:19.340291 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:19.340267    4447 cpu_manager.go:414] "containerMap: removing container" podUID="64782f21-9364-4cd2-9731-cd27f2ba2c29" containerName="console"
Oct 31 03:23:19.340291 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:19.340272    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64782f21-9364-4cd2-9731-cd27f2ba2c29" containerName="console"
Oct 31 03:23:19.340379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:19.340320    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b7b96f3-2670-4021-a387-15f089d2f1d1" containerName="console"
Oct 31 03:23:19.340379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:19.340331    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="64782f21-9364-4cd2-9731-cd27f2ba2c29" containerName="console"
Oct 31 03:23:19.342288 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:19.342268    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql"
Oct 31 03:23:19.400341 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:19.400310    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-q5rc9\""
Oct 31 03:23:19.442464 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:19.442431    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a682781-0d16-480f-9610-a8bfb1abf481-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql\" (UID: \"7a682781-0d16-480f-9610-a8bfb1abf481\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql"
Oct 31 03:23:19.442623 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:19.442474    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bqb2\" (UniqueName: \"kubernetes.io/projected/7a682781-0d16-480f-9610-a8bfb1abf481-kube-api-access-5bqb2\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql\" (UID: \"7a682781-0d16-480f-9610-a8bfb1abf481\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql"
Oct 31 03:23:19.442623 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:19.442585    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a682781-0d16-480f-9610-a8bfb1abf481-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql\" (UID: \"7a682781-0d16-480f-9610-a8bfb1abf481\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql"
Oct 31 03:23:19.450558 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:19.450534    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 03:23:19.450558 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:19.450536    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 03:23:19.451770 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:19.451743    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql"]
Oct 31 03:23:19.543965 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:19.543931    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a682781-0d16-480f-9610-a8bfb1abf481-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql\" (UID: \"7a682781-0d16-480f-9610-a8bfb1abf481\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql"
Oct 31 03:23:19.544152 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:19.543994    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a682781-0d16-480f-9610-a8bfb1abf481-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql\" (UID: \"7a682781-0d16-480f-9610-a8bfb1abf481\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql"
Oct 31 03:23:19.544152 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:19.544021    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bqb2\" (UniqueName: \"kubernetes.io/projected/7a682781-0d16-480f-9610-a8bfb1abf481-kube-api-access-5bqb2\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql\" (UID: \"7a682781-0d16-480f-9610-a8bfb1abf481\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql"
Oct 31 03:23:19.544378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:19.544358    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a682781-0d16-480f-9610-a8bfb1abf481-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql\" (UID: \"7a682781-0d16-480f-9610-a8bfb1abf481\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql"
Oct 31 03:23:19.544378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:19.544369    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a682781-0d16-480f-9610-a8bfb1abf481-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql\" (UID: \"7a682781-0d16-480f-9610-a8bfb1abf481\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql"
Oct 31 03:23:19.605333 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:19.605257    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bqb2\" (UniqueName: \"kubernetes.io/projected/7a682781-0d16-480f-9610-a8bfb1abf481-kube-api-access-5bqb2\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql\" (UID: \"7a682781-0d16-480f-9610-a8bfb1abf481\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql"
Oct 31 03:23:19.654190 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:19.654155    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql"
Oct 31 03:23:20.023535 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:20.023500    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql"]
Oct 31 03:23:20.056339 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:23:20.056306    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a682781_0d16_480f_9610_a8bfb1abf481.slice/crio-21c409b7338f0157e81c538007c2b22fe13de4da5938d1bc4a82d2fdd7f08182 WatchSource:0}: Error finding container 21c409b7338f0157e81c538007c2b22fe13de4da5938d1bc4a82d2fdd7f08182: Status 404 returned error can't find the container with id 21c409b7338f0157e81c538007c2b22fe13de4da5938d1bc4a82d2fdd7f08182
Oct 31 03:23:20.568032 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:20.568002    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql" event={"ID":"7a682781-0d16-480f-9610-a8bfb1abf481","Type":"ContainerStarted","Data":"21c409b7338f0157e81c538007c2b22fe13de4da5938d1bc4a82d2fdd7f08182"}
Oct 31 03:23:20.811906 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:20.811858    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb"]
Oct 31 03:23:20.814809 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:20.814792    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb"
Oct 31 03:23:20.859435 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:20.859407    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c3b7b38-a190-451c-bd75-bc48ed07b5e5-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb\" (UID: \"2c3b7b38-a190-451c-bd75-bc48ed07b5e5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb"
Oct 31 03:23:20.859607 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:20.859464    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gk8n\" (UniqueName: \"kubernetes.io/projected/2c3b7b38-a190-451c-bd75-bc48ed07b5e5-kube-api-access-4gk8n\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb\" (UID: \"2c3b7b38-a190-451c-bd75-bc48ed07b5e5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb"
Oct 31 03:23:20.859607 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:20.859544    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c3b7b38-a190-451c-bd75-bc48ed07b5e5-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb\" (UID: \"2c3b7b38-a190-451c-bd75-bc48ed07b5e5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb"
Oct 31 03:23:20.917466 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:20.917433    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-zxcxh\""
Oct 31 03:23:20.918827 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:20.918805    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb"]
Oct 31 03:23:20.960148 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:20.960111    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gk8n\" (UniqueName: \"kubernetes.io/projected/2c3b7b38-a190-451c-bd75-bc48ed07b5e5-kube-api-access-4gk8n\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb\" (UID: \"2c3b7b38-a190-451c-bd75-bc48ed07b5e5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb"
Oct 31 03:23:20.960148 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:20.960151    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c3b7b38-a190-451c-bd75-bc48ed07b5e5-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb\" (UID: \"2c3b7b38-a190-451c-bd75-bc48ed07b5e5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb"
Oct 31 03:23:20.960351 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:20.960212    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c3b7b38-a190-451c-bd75-bc48ed07b5e5-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb\" (UID: \"2c3b7b38-a190-451c-bd75-bc48ed07b5e5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb"
Oct 31 03:23:20.960557 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:20.960541    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c3b7b38-a190-451c-bd75-bc48ed07b5e5-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb\" (UID: \"2c3b7b38-a190-451c-bd75-bc48ed07b5e5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb"
Oct 31 03:23:20.960608 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:20.960590    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c3b7b38-a190-451c-bd75-bc48ed07b5e5-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb\" (UID: \"2c3b7b38-a190-451c-bd75-bc48ed07b5e5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb"
Oct 31 03:23:21.021554 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:21.021520    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gk8n\" (UniqueName: \"kubernetes.io/projected/2c3b7b38-a190-451c-bd75-bc48ed07b5e5-kube-api-access-4gk8n\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb\" (UID: \"2c3b7b38-a190-451c-bd75-bc48ed07b5e5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb"
Oct 31 03:23:21.126657 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:21.126572    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb"
Oct 31 03:23:21.508794 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:21.508734    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb"]
Oct 31 03:23:21.511688 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:23:21.511650    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3b7b38_a190_451c_bd75_bc48ed07b5e5.slice/crio-041d61a721642787286aec6a3604b181d489267212f4bb5a1c42b4b52376a87c WatchSource:0}: Error finding container 041d61a721642787286aec6a3604b181d489267212f4bb5a1c42b4b52376a87c: Status 404 returned error can't find the container with id 041d61a721642787286aec6a3604b181d489267212f4bb5a1c42b4b52376a87c
Oct 31 03:23:21.573436 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:21.573397    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb" event={"ID":"2c3b7b38-a190-451c-bd75-bc48ed07b5e5","Type":"ContainerStarted","Data":"041d61a721642787286aec6a3604b181d489267212f4bb5a1c42b4b52376a87c"}
Oct 31 03:23:23.239568 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:23.239534    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk"]
Oct 31 03:23:23.242590 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:23.242559    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk"
Oct 31 03:23:23.280459 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:23.280423    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v4xm\" (UniqueName: \"kubernetes.io/projected/02a07d7a-d488-4f80-8d96-323b749ffdbd-kube-api-access-2v4xm\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk\" (UID: \"02a07d7a-d488-4f80-8d96-323b749ffdbd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk"
Oct 31 03:23:23.280623 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:23.280485    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02a07d7a-d488-4f80-8d96-323b749ffdbd-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk\" (UID: \"02a07d7a-d488-4f80-8d96-323b749ffdbd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk"
Oct 31 03:23:23.280623 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:23.280537    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02a07d7a-d488-4f80-8d96-323b749ffdbd-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk\" (UID: \"02a07d7a-d488-4f80-8d96-323b749ffdbd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk"
Oct 31 03:23:23.345201 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:23.345169    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-gjcqp\""
Oct 31 03:23:23.346826 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:23.346791    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk"]
Oct 31 03:23:23.381762 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:23.381725    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v4xm\" (UniqueName: \"kubernetes.io/projected/02a07d7a-d488-4f80-8d96-323b749ffdbd-kube-api-access-2v4xm\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk\" (UID: \"02a07d7a-d488-4f80-8d96-323b749ffdbd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk"
Oct 31 03:23:23.381980 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:23.381791    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02a07d7a-d488-4f80-8d96-323b749ffdbd-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk\" (UID: \"02a07d7a-d488-4f80-8d96-323b749ffdbd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk"
Oct 31 03:23:23.381980 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:23.381843    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02a07d7a-d488-4f80-8d96-323b749ffdbd-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk\" (UID: \"02a07d7a-d488-4f80-8d96-323b749ffdbd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk"
Oct 31 03:23:23.382205 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:23.382180    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02a07d7a-d488-4f80-8d96-323b749ffdbd-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk\" (UID: \"02a07d7a-d488-4f80-8d96-323b749ffdbd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk"
Oct 31 03:23:23.382271 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:23.382231    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02a07d7a-d488-4f80-8d96-323b749ffdbd-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk\" (UID: \"02a07d7a-d488-4f80-8d96-323b749ffdbd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk"
Oct 31 03:23:23.444617 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:23.444584    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v4xm\" (UniqueName: \"kubernetes.io/projected/02a07d7a-d488-4f80-8d96-323b749ffdbd-kube-api-access-2v4xm\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk\" (UID: \"02a07d7a-d488-4f80-8d96-323b749ffdbd\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk"
Oct 31 03:23:23.558984 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:23.558909    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk"
Oct 31 03:23:23.942842 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:23.942755    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk"]
Oct 31 03:23:23.947140 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:23:23.947071    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a07d7a_d488_4f80_8d96_323b749ffdbd.slice/crio-f83655603f468b535f68b6e676d8d4fcc9256de2a5b6e061ffeb437b974f44d6 WatchSource:0}: Error finding container f83655603f468b535f68b6e676d8d4fcc9256de2a5b6e061ffeb437b974f44d6: Status 404 returned error can't find the container with id f83655603f468b535f68b6e676d8d4fcc9256de2a5b6e061ffeb437b974f44d6
Oct 31 03:23:24.591770 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.591735    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk" event={"ID":"02a07d7a-d488-4f80-8d96-323b749ffdbd","Type":"ContainerStarted","Data":"f83655603f468b535f68b6e676d8d4fcc9256de2a5b6e061ffeb437b974f44d6"}
Oct 31 03:23:24.709036 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.708999    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm"]
Oct 31 03:23:24.714538 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.714510    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm"
Oct 31 03:23:24.796171 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.796122    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjsv8\" (UniqueName: \"kubernetes.io/projected/59be87b9-6885-4ab3-8705-580a8ba49625-kube-api-access-jjsv8\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm\" (UID: \"59be87b9-6885-4ab3-8705-580a8ba49625\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm"
Oct 31 03:23:24.796340 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.796311    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59be87b9-6885-4ab3-8705-580a8ba49625-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm\" (UID: \"59be87b9-6885-4ab3-8705-580a8ba49625\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm"
Oct 31 03:23:24.796421 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.796341    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59be87b9-6885-4ab3-8705-580a8ba49625-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm\" (UID: \"59be87b9-6885-4ab3-8705-580a8ba49625\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm"
Oct 31 03:23:24.825158 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.819825    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-mhsmw\""
Oct 31 03:23:24.825158 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.821489    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm"]
Oct 31 03:23:24.825158 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.822961    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj"]
Oct 31 03:23:24.827957 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.827933    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj"
Oct 31 03:23:24.885711 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.882542    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-qx7zn\""
Oct 31 03:23:24.885711 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.884309    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj"]
Oct 31 03:23:24.897466 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.897432    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2298be1-df21-4e9b-b1ef-75559fb24494-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj\" (UID: \"e2298be1-df21-4e9b-b1ef-75559fb24494\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj"
Oct 31 03:23:24.897633 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.897501    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpmv6\" (UniqueName: \"kubernetes.io/projected/e2298be1-df21-4e9b-b1ef-75559fb24494-kube-api-access-vpmv6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj\" (UID: \"e2298be1-df21-4e9b-b1ef-75559fb24494\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj"
Oct 31 03:23:24.897633 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.897534    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2298be1-df21-4e9b-b1ef-75559fb24494-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj\" (UID: \"e2298be1-df21-4e9b-b1ef-75559fb24494\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj"
Oct 31 03:23:24.897633 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.897566    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59be87b9-6885-4ab3-8705-580a8ba49625-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm\" (UID: \"59be87b9-6885-4ab3-8705-580a8ba49625\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm"
Oct 31 03:23:24.897633 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.897606    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59be87b9-6885-4ab3-8705-580a8ba49625-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm\" (UID: \"59be87b9-6885-4ab3-8705-580a8ba49625\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm"
Oct 31 03:23:24.897849 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.897652    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjsv8\" (UniqueName: \"kubernetes.io/projected/59be87b9-6885-4ab3-8705-580a8ba49625-kube-api-access-jjsv8\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm\" (UID: \"59be87b9-6885-4ab3-8705-580a8ba49625\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm"
Oct 31 03:23:24.898329 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.898306    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59be87b9-6885-4ab3-8705-580a8ba49625-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm\" (UID: \"59be87b9-6885-4ab3-8705-580a8ba49625\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm"
Oct 31 03:23:24.898589 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.898571    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59be87b9-6885-4ab3-8705-580a8ba49625-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm\" (UID: \"59be87b9-6885-4ab3-8705-580a8ba49625\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm"
Oct 31 03:23:24.935052 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.935021    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk"]
Oct 31 03:23:24.939910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.939870    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk"
Oct 31 03:23:24.960545 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.960508    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjsv8\" (UniqueName: \"kubernetes.io/projected/59be87b9-6885-4ab3-8705-580a8ba49625-kube-api-access-jjsv8\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm\" (UID: \"59be87b9-6885-4ab3-8705-580a8ba49625\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm"
Oct 31 03:23:24.960915 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.960870    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk"]
Oct 31 03:23:24.993088 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.993032    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-s7fnb\""
Oct 31 03:23:24.998706 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.998666    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pf9l\" (UniqueName: \"kubernetes.io/projected/206a4f55-822f-4ec0-ac3d-4ff8df04f2dd-kube-api-access-4pf9l\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk\" (UID: \"206a4f55-822f-4ec0-ac3d-4ff8df04f2dd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk"
Oct 31 03:23:24.998870 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.998814    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/206a4f55-822f-4ec0-ac3d-4ff8df04f2dd-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk\" (UID: \"206a4f55-822f-4ec0-ac3d-4ff8df04f2dd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk"
Oct 31 03:23:24.998985 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.998930    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2298be1-df21-4e9b-b1ef-75559fb24494-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj\" (UID: \"e2298be1-df21-4e9b-b1ef-75559fb24494\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj"
Oct 31 03:23:24.999038 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.998980    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpmv6\" (UniqueName: \"kubernetes.io/projected/e2298be1-df21-4e9b-b1ef-75559fb24494-kube-api-access-vpmv6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj\" (UID: \"e2298be1-df21-4e9b-b1ef-75559fb24494\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj"
Oct 31 03:23:24.999038 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.999007    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/206a4f55-822f-4ec0-ac3d-4ff8df04f2dd-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk\" (UID: \"206a4f55-822f-4ec0-ac3d-4ff8df04f2dd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk"
Oct 31 03:23:24.999122 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.999038    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2298be1-df21-4e9b-b1ef-75559fb24494-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj\" (UID: \"e2298be1-df21-4e9b-b1ef-75559fb24494\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj"
Oct 31 03:23:24.999466 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.999445    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2298be1-df21-4e9b-b1ef-75559fb24494-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj\" (UID: \"e2298be1-df21-4e9b-b1ef-75559fb24494\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj"
Oct 31 03:23:24.999685 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:24.999670    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2298be1-df21-4e9b-b1ef-75559fb24494-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj\" (UID: \"e2298be1-df21-4e9b-b1ef-75559fb24494\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj"
Oct 31 03:23:25.002984 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.002959    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5"]
Oct 31 03:23:25.007649 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.007623    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5"
Oct 31 03:23:25.039248 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.039203    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm"
Oct 31 03:23:25.062070 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.062025    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-j276r\""
Oct 31 03:23:25.064459 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.064429    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpmv6\" (UniqueName: \"kubernetes.io/projected/e2298be1-df21-4e9b-b1ef-75559fb24494-kube-api-access-vpmv6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj\" (UID: \"e2298be1-df21-4e9b-b1ef-75559fb24494\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj"
Oct 31 03:23:25.065663 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.065565    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5"]
Oct 31 03:23:25.100492 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.100460    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ef89243-9c18-42f8-92cc-e04cd350a4dc-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5\" (UID: \"1ef89243-9c18-42f8-92cc-e04cd350a4dc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5"
Oct 31 03:23:25.100652 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.100507    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/206a4f55-822f-4ec0-ac3d-4ff8df04f2dd-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk\" (UID: \"206a4f55-822f-4ec0-ac3d-4ff8df04f2dd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk"
Oct 31 03:23:25.101579 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.100763    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pf9l\" (UniqueName: \"kubernetes.io/projected/206a4f55-822f-4ec0-ac3d-4ff8df04f2dd-kube-api-access-4pf9l\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk\" (UID: \"206a4f55-822f-4ec0-ac3d-4ff8df04f2dd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk"
Oct 31 03:23:25.101579 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.100806    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ef89243-9c18-42f8-92cc-e04cd350a4dc-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5\" (UID: \"1ef89243-9c18-42f8-92cc-e04cd350a4dc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5"
Oct 31 03:23:25.101579 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.100836    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4scw\" (UniqueName: \"kubernetes.io/projected/1ef89243-9c18-42f8-92cc-e04cd350a4dc-kube-api-access-p4scw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5\" (UID: \"1ef89243-9c18-42f8-92cc-e04cd350a4dc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5"
Oct 31 03:23:25.101579 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.100917    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/206a4f55-822f-4ec0-ac3d-4ff8df04f2dd-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk\" (UID: \"206a4f55-822f-4ec0-ac3d-4ff8df04f2dd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk"
Oct 31 03:23:25.101579 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.101236    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/206a4f55-822f-4ec0-ac3d-4ff8df04f2dd-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk\" (UID: \"206a4f55-822f-4ec0-ac3d-4ff8df04f2dd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk"
Oct 31 03:23:25.102387 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.102353    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/206a4f55-822f-4ec0-ac3d-4ff8df04f2dd-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk\" (UID: \"206a4f55-822f-4ec0-ac3d-4ff8df04f2dd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk"
Oct 31 03:23:25.149985 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.149663    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj"
Oct 31 03:23:25.163501 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.163302    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pf9l\" (UniqueName: \"kubernetes.io/projected/206a4f55-822f-4ec0-ac3d-4ff8df04f2dd-kube-api-access-4pf9l\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk\" (UID: \"206a4f55-822f-4ec0-ac3d-4ff8df04f2dd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk"
Oct 31 03:23:25.202313 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.202277    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ef89243-9c18-42f8-92cc-e04cd350a4dc-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5\" (UID: \"1ef89243-9c18-42f8-92cc-e04cd350a4dc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5"
Oct 31 03:23:25.202503 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.202362    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ef89243-9c18-42f8-92cc-e04cd350a4dc-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5\" (UID: \"1ef89243-9c18-42f8-92cc-e04cd350a4dc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5"
Oct 31 03:23:25.202503 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.202392    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4scw\" (UniqueName: \"kubernetes.io/projected/1ef89243-9c18-42f8-92cc-e04cd350a4dc-kube-api-access-p4scw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5\" (UID: \"1ef89243-9c18-42f8-92cc-e04cd350a4dc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5"
Oct 31 03:23:25.202735 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.202711    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ef89243-9c18-42f8-92cc-e04cd350a4dc-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5\" (UID: \"1ef89243-9c18-42f8-92cc-e04cd350a4dc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5"
Oct 31 03:23:25.202812 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.202750    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ef89243-9c18-42f8-92cc-e04cd350a4dc-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5\" (UID: \"1ef89243-9c18-42f8-92cc-e04cd350a4dc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5"
Oct 31 03:23:25.262248 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.261923    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk"
Oct 31 03:23:25.271179 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.271149    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4scw\" (UniqueName: \"kubernetes.io/projected/1ef89243-9c18-42f8-92cc-e04cd350a4dc-kube-api-access-p4scw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5\" (UID: \"1ef89243-9c18-42f8-92cc-e04cd350a4dc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5"
Oct 31 03:23:25.347653 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.347403    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5"
Oct 31 03:23:25.440364 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.440136    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm"]
Oct 31 03:23:25.449609 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:23:25.449578    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59be87b9_6885_4ab3_8705_580a8ba49625.slice/crio-11887a149e80b9fa5342da9a001057300c3d2e0a78e12943b91baaea11741597 WatchSource:0}: Error finding container 11887a149e80b9fa5342da9a001057300c3d2e0a78e12943b91baaea11741597: Status 404 returned error can't find the container with id 11887a149e80b9fa5342da9a001057300c3d2e0a78e12943b91baaea11741597
Oct 31 03:23:25.552702 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.552656    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj"]
Oct 31 03:23:25.555930 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:23:25.555879    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2298be1_df21_4e9b_b1ef_75559fb24494.slice/crio-ed61e90488b3827dfeb112aeeee19df04a133c923df7d2b90404f47ff9db1878 WatchSource:0}: Error finding container ed61e90488b3827dfeb112aeeee19df04a133c923df7d2b90404f47ff9db1878: Status 404 returned error can't find the container with id ed61e90488b3827dfeb112aeeee19df04a133c923df7d2b90404f47ff9db1878
Oct 31 03:23:25.599027 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.598934    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj" event={"ID":"e2298be1-df21-4e9b-b1ef-75559fb24494","Type":"ContainerStarted","Data":"ed61e90488b3827dfeb112aeeee19df04a133c923df7d2b90404f47ff9db1878"}
Oct 31 03:23:25.602065 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.602001    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm" event={"ID":"59be87b9-6885-4ab3-8705-580a8ba49625","Type":"ContainerStarted","Data":"11887a149e80b9fa5342da9a001057300c3d2e0a78e12943b91baaea11741597"}
Oct 31 03:23:25.662799 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.662683    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk"]
Oct 31 03:23:25.666192 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:23:25.666157    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod206a4f55_822f_4ec0_ac3d_4ff8df04f2dd.slice/crio-39e807dc5036b0b33a2722477a653fb9bd35a435f29480441347f99f5921ccbc WatchSource:0}: Error finding container 39e807dc5036b0b33a2722477a653fb9bd35a435f29480441347f99f5921ccbc: Status 404 returned error can't find the container with id 39e807dc5036b0b33a2722477a653fb9bd35a435f29480441347f99f5921ccbc
Oct 31 03:23:25.749486 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:25.749453    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5"]
Oct 31 03:23:25.753990 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:23:25.753948    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef89243_9c18_42f8_92cc_e04cd350a4dc.slice/crio-3d71978f35ef8f7a99e6708daab98b4ae0c8fb89005027e3782747d1a4dc8e17 WatchSource:0}: Error finding container 3d71978f35ef8f7a99e6708daab98b4ae0c8fb89005027e3782747d1a4dc8e17: Status 404 returned error can't find the container with id 3d71978f35ef8f7a99e6708daab98b4ae0c8fb89005027e3782747d1a4dc8e17
Oct 31 03:23:26.611939 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.611898    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5" event={"ID":"1ef89243-9c18-42f8-92cc-e04cd350a4dc","Type":"ContainerStarted","Data":"3d71978f35ef8f7a99e6708daab98b4ae0c8fb89005027e3782747d1a4dc8e17"}
Oct 31 03:23:26.613611 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.613587    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk" event={"ID":"206a4f55-822f-4ec0-ac3d-4ff8df04f2dd","Type":"ContainerStarted","Data":"39e807dc5036b0b33a2722477a653fb9bd35a435f29480441347f99f5921ccbc"}
Oct 31 03:23:26.641121 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.641087    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"]
Oct 31 03:23:26.646140 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.646108    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.699538 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.699500    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 03:23:26.716925 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.716865    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c539e6bf-2bd3-482c-9881-5c3fd79143e2-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.717109 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.716980    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nql9\" (UniqueName: \"kubernetes.io/projected/c539e6bf-2bd3-482c-9881-5c3fd79143e2-kube-api-access-9nql9\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.717109 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.717046    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c539e6bf-2bd3-482c-9881-5c3fd79143e2-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.717109 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.717075    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c539e6bf-2bd3-482c-9881-5c3fd79143e2-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.717265 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.717115    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c539e6bf-2bd3-482c-9881-5c3fd79143e2-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.717265 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.717156    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c539e6bf-2bd3-482c-9881-5c3fd79143e2-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.717265 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.717203    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c539e6bf-2bd3-482c-9881-5c3fd79143e2-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.717265 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.717242    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c539e6bf-2bd3-482c-9881-5c3fd79143e2-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.717456 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.717285    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c539e6bf-2bd3-482c-9881-5c3fd79143e2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.734632 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.732178    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 03:23:26.734632 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.732245    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 03:23:26.748609 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.748520    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-f2w6z\""
Oct 31 03:23:26.750962 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.750774    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"]
Oct 31 03:23:26.819912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.818357    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c539e6bf-2bd3-482c-9881-5c3fd79143e2-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.819912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.818413    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c539e6bf-2bd3-482c-9881-5c3fd79143e2-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.819912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.818451    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c539e6bf-2bd3-482c-9881-5c3fd79143e2-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.819912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.818472    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c539e6bf-2bd3-482c-9881-5c3fd79143e2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.819912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.818494    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c539e6bf-2bd3-482c-9881-5c3fd79143e2-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.819912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.818527    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nql9\" (UniqueName: \"kubernetes.io/projected/c539e6bf-2bd3-482c-9881-5c3fd79143e2-kube-api-access-9nql9\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.819912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.818570    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c539e6bf-2bd3-482c-9881-5c3fd79143e2-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.819912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.818592    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c539e6bf-2bd3-482c-9881-5c3fd79143e2-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.819912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.818635    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c539e6bf-2bd3-482c-9881-5c3fd79143e2-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.819912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.819101    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c539e6bf-2bd3-482c-9881-5c3fd79143e2-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.819912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.819769    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c539e6bf-2bd3-482c-9881-5c3fd79143e2-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.822370 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.821031    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c539e6bf-2bd3-482c-9881-5c3fd79143e2-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.822370 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.821668    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c539e6bf-2bd3-482c-9881-5c3fd79143e2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.822370 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.821921    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c539e6bf-2bd3-482c-9881-5c3fd79143e2-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.825679 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.825619    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c539e6bf-2bd3-482c-9881-5c3fd79143e2-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.826324 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.826303    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c539e6bf-2bd3-482c-9881-5c3fd79143e2-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.890406 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.890321    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c539e6bf-2bd3-482c-9881-5c3fd79143e2-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.892947 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.892915    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nql9\" (UniqueName: \"kubernetes.io/projected/c539e6bf-2bd3-482c-9881-5c3fd79143e2-kube-api-access-9nql9\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v\" (UID: \"c539e6bf-2bd3-482c-9881-5c3fd79143e2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:26.956370 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.956337    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"]
Oct 31 03:23:26.963135 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.962248    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:26.964901 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:26.964649    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:27.018287 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.018254    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4td2m\""
Oct 31 03:23:27.025873 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.025329    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.025873 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.025370    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.025873 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.025422    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c79e465-b30e-482a-bccb-61f5bcb4d910-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.025873 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.025452    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.025873 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.025489    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.025873 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.025525    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.025873 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.025595    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.025873 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.025636    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.025873 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.025670    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mn84\" (UniqueName: \"kubernetes.io/projected/7c79e465-b30e-482a-bccb-61f5bcb4d910-kube-api-access-8mn84\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.039295 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.039263    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"]
Oct 31 03:23:27.127383 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.127342    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mn84\" (UniqueName: \"kubernetes.io/projected/7c79e465-b30e-482a-bccb-61f5bcb4d910-kube-api-access-8mn84\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.127540 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.127420    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.127540 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.127442    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.127540 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.127489    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c79e465-b30e-482a-bccb-61f5bcb4d910-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.127540 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.127521    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.127751 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.127554    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.127751 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.127583    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.127751 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.127633    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.127751 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.127671    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.129912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.128581    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c79e465-b30e-482a-bccb-61f5bcb4d910-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.129912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.129085    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.129912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.129773    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.130142 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.130075    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.130328 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.130282    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.132795 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.132770    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.139186 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.139130    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.191414 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.191331    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mn84\" (UniqueName: \"kubernetes.io/projected/7c79e465-b30e-482a-bccb-61f5bcb4d910-kube-api-access-8mn84\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.193962 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.193911    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-9qtng\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.282337 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.282032    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:27.465465 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.465427    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"]
Oct 31 03:23:27.471363 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:23:27.471309    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc539e6bf_2bd3_482c_9881_5c3fd79143e2.slice/crio-4e2e57c4a24ce9c25e38c5f858899deeea5e6c42c9bfc4b6c5ffbc116d4281cc WatchSource:0}: Error finding container 4e2e57c4a24ce9c25e38c5f858899deeea5e6c42c9bfc4b6c5ffbc116d4281cc: Status 404 returned error can't find the container with id 4e2e57c4a24ce9c25e38c5f858899deeea5e6c42c9bfc4b6c5ffbc116d4281cc
Oct 31 03:23:27.622037 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.621952    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v" event={"ID":"c539e6bf-2bd3-482c-9881-5c3fd79143e2","Type":"ContainerStarted","Data":"4e2e57c4a24ce9c25e38c5f858899deeea5e6c42c9bfc4b6c5ffbc116d4281cc"}
Oct 31 03:23:27.727536 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:27.727213    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"]
Oct 31 03:23:27.735735 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:23:27.735678    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c79e465_b30e_482a_bccb_61f5bcb4d910.slice/crio-c285028bfa50a6de1a747ec89c3654785848fbc4ecfc9c3e40056f10068b3b4b WatchSource:0}: Error finding container c285028bfa50a6de1a747ec89c3654785848fbc4ecfc9c3e40056f10068b3b4b: Status 404 returned error can't find the container with id c285028bfa50a6de1a747ec89c3654785848fbc4ecfc9c3e40056f10068b3b4b
Oct 31 03:23:28.639176 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:28.639136    4447 generic.go:356] "Generic (PLEG): container finished" podID="7a682781-0d16-480f-9610-a8bfb1abf481" containerID="9f5f4dbae0be75a8979523ef65486032e08c91223701d911c686f3dc2ebc413a" exitCode=0
Oct 31 03:23:28.639645 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:28.639320    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql" event={"ID":"7a682781-0d16-480f-9610-a8bfb1abf481","Type":"ContainerDied","Data":"9f5f4dbae0be75a8979523ef65486032e08c91223701d911c686f3dc2ebc413a"}
Oct 31 03:23:28.643799 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:28.643701    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng" event={"ID":"7c79e465-b30e-482a-bccb-61f5bcb4d910","Type":"ContainerStarted","Data":"c285028bfa50a6de1a747ec89c3654785848fbc4ecfc9c3e40056f10068b3b4b"}
Oct 31 03:23:32.688540 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:32.687824    4447 generic.go:356] "Generic (PLEG): container finished" podID="02a07d7a-d488-4f80-8d96-323b749ffdbd" containerID="bed43883393a9c030786fdb7c373561734d9031c15eaabcca5ad73500dc773fa" exitCode=0
Oct 31 03:23:32.688540 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:32.687941    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk" event={"ID":"02a07d7a-d488-4f80-8d96-323b749ffdbd","Type":"ContainerDied","Data":"bed43883393a9c030786fdb7c373561734d9031c15eaabcca5ad73500dc773fa"}
Oct 31 03:23:32.693432 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:32.692276    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb" event={"ID":"2c3b7b38-a190-451c-bd75-bc48ed07b5e5","Type":"ContainerStarted","Data":"aa6022d8d8b3622d4a3f9af85bd89317c534f18c75a85a6c42d87ae78133f52e"}
Oct 31 03:23:33.702696 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:33.702659    4447 generic.go:356] "Generic (PLEG): container finished" podID="2c3b7b38-a190-451c-bd75-bc48ed07b5e5" containerID="aa6022d8d8b3622d4a3f9af85bd89317c534f18c75a85a6c42d87ae78133f52e" exitCode=0
Oct 31 03:23:33.703178 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:33.702749    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb" event={"ID":"2c3b7b38-a190-451c-bd75-bc48ed07b5e5","Type":"ContainerDied","Data":"aa6022d8d8b3622d4a3f9af85bd89317c534f18c75a85a6c42d87ae78133f52e"}
Oct 31 03:23:34.712218 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:34.712176    4447 generic.go:356] "Generic (PLEG): container finished" podID="2c3b7b38-a190-451c-bd75-bc48ed07b5e5" containerID="a05b6aabc9acd66773bd830749c2af4722c4980e17c0828601bdc7c8731fbdf9" exitCode=0
Oct 31 03:23:34.712651 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:34.712296    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb" event={"ID":"2c3b7b38-a190-451c-bd75-bc48ed07b5e5","Type":"ContainerDied","Data":"a05b6aabc9acd66773bd830749c2af4722c4980e17c0828601bdc7c8731fbdf9"}
Oct 31 03:23:34.719334 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:34.719303    4447 generic.go:356] "Generic (PLEG): container finished" podID="59be87b9-6885-4ab3-8705-580a8ba49625" containerID="6c458d41f429136b764fee91d5c42376f2b3a7542d17a752262cfaa4a2251f4e" exitCode=0
Oct 31 03:23:34.719445 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:34.719405    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm" event={"ID":"59be87b9-6885-4ab3-8705-580a8ba49625","Type":"ContainerDied","Data":"6c458d41f429136b764fee91d5c42376f2b3a7542d17a752262cfaa4a2251f4e"}
Oct 31 03:23:34.724151 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:34.724120    4447 generic.go:356] "Generic (PLEG): container finished" podID="e2298be1-df21-4e9b-b1ef-75559fb24494" containerID="c9b4f75ae0bdefaa6ae56a4c2bad85fb13d5d82fe71e0ae4ee7e989ecfbbebf7" exitCode=0
Oct 31 03:23:34.724275 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:34.724169    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj" event={"ID":"e2298be1-df21-4e9b-b1ef-75559fb24494","Type":"ContainerDied","Data":"c9b4f75ae0bdefaa6ae56a4c2bad85fb13d5d82fe71e0ae4ee7e989ecfbbebf7"}
Oct 31 03:23:35.327183 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:35.327150    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:23:35.327307 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:35.327241    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:23:35.327307 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:35.327283    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:23:35.432752 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:35.432707    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:23:35.432849 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:35.432791    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:23:35.432849 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:35.432820    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:23:35.732241 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:35.732203    4447 generic.go:356] "Generic (PLEG): container finished" podID="1ef89243-9c18-42f8-92cc-e04cd350a4dc" containerID="ca8fe0acd33b52b15de000f56dcb455f52e0d85d3ad7fa225ebc54d7b775d776" exitCode=0
Oct 31 03:23:35.732738 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:35.732261    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5" event={"ID":"1ef89243-9c18-42f8-92cc-e04cd350a4dc","Type":"ContainerDied","Data":"ca8fe0acd33b52b15de000f56dcb455f52e0d85d3ad7fa225ebc54d7b775d776"}
Oct 31 03:23:35.735026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:35.734469    4447 generic.go:356] "Generic (PLEG): container finished" podID="206a4f55-822f-4ec0-ac3d-4ff8df04f2dd" containerID="357716df6d68da8e9fadc9ae65922fb3a3ff5bdad6f131e5a7e84ef6469c5336" exitCode=0
Oct 31 03:23:35.735026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:35.734574    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk" event={"ID":"206a4f55-822f-4ec0-ac3d-4ff8df04f2dd","Type":"ContainerDied","Data":"357716df6d68da8e9fadc9ae65922fb3a3ff5bdad6f131e5a7e84ef6469c5336"}
Oct 31 03:23:35.737416 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:35.737392    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng" event={"ID":"7c79e465-b30e-482a-bccb-61f5bcb4d910","Type":"ContainerStarted","Data":"c7653d66d44d682f4e02f0195c77262823daf9c3dc1e17068cbe803a7d626110"}
Oct 31 03:23:35.739765 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:35.739743    4447 generic.go:356] "Generic (PLEG): container finished" podID="7a682781-0d16-480f-9610-a8bfb1abf481" containerID="ac702811a02c25664a784d21ff0eb50dd2a6c5c4f5b5d899f5037078f7817aa0" exitCode=0
Oct 31 03:23:35.739938 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:35.739851    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql" event={"ID":"7a682781-0d16-480f-9610-a8bfb1abf481","Type":"ContainerDied","Data":"ac702811a02c25664a784d21ff0eb50dd2a6c5c4f5b5d899f5037078f7817aa0"}
Oct 31 03:23:35.742010 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:35.741984    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v" event={"ID":"c539e6bf-2bd3-482c-9881-5c3fd79143e2","Type":"ContainerStarted","Data":"30b9f0b429307637e18d310cafc0e79588f1dc56804f04b25518f4abe084c0d9"}
Oct 31 03:23:35.965728 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:35.965696    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:36.099976 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:36.098241    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng" podStartSLOduration=2.51360464 podStartE2EDuration="10.098222181s" totalImagesPullingTime="7.584617541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:35.974088273 +0000 UTC m=+241.485070650" watchObservedRunningTime="2026-10-31 03:23:36.098222181 +0000 UTC m=+241.609204563"
Oct 31 03:23:36.099976 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:36.098744    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v" podStartSLOduration=2.14179072 podStartE2EDuration="10.098733535s" totalImagesPullingTime="7.956942815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:36.094958514 +0000 UTC m=+241.605940891" watchObservedRunningTime="2026-10-31 03:23:36.098733535 +0000 UTC m=+241.609715915"
Oct 31 03:23:36.282741 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:36.282705    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:36.288059 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:36.288030    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:36.748877 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:36.748821    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:36.750196 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:36.750172    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:23:36.971115 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:36.971087    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:37.755714 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:37.755688    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:37.757562 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:37.757312    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v"
Oct 31 03:23:38.762523 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:38.762491    4447 generic.go:356] "Generic (PLEG): container finished" podID="59be87b9-6885-4ab3-8705-580a8ba49625" containerID="fee972a43cb138edae81cbd027639b8c77ca3b444b888e89787111f93e50da7f" exitCode=0
Oct 31 03:23:38.763026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:38.762586    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm" event={"ID":"59be87b9-6885-4ab3-8705-580a8ba49625","Type":"ContainerDied","Data":"fee972a43cb138edae81cbd027639b8c77ca3b444b888e89787111f93e50da7f"}
Oct 31 03:23:38.765657 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:38.765632    4447 generic.go:356] "Generic (PLEG): container finished" podID="02a07d7a-d488-4f80-8d96-323b749ffdbd" containerID="b696d13a6f51ae0a21f530cbd681388a5d255bc775486db7abd7c6677ccf9ede" exitCode=0
Oct 31 03:23:38.765792 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:38.765772    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk" event={"ID":"02a07d7a-d488-4f80-8d96-323b749ffdbd","Type":"ContainerDied","Data":"b696d13a6f51ae0a21f530cbd681388a5d255bc775486db7abd7c6677ccf9ede"}
Oct 31 03:23:39.778129 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:39.778090    4447 generic.go:356] "Generic (PLEG): container finished" podID="e2298be1-df21-4e9b-b1ef-75559fb24494" containerID="db04963768b1ba6ee3f89493115c7dc5a121110274a5182f7f24b506d22306c5" exitCode=0
Oct 31 03:23:39.779935 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:39.779478    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj" event={"ID":"e2298be1-df21-4e9b-b1ef-75559fb24494","Type":"ContainerDied","Data":"db04963768b1ba6ee3f89493115c7dc5a121110274a5182f7f24b506d22306c5"}
Oct 31 03:23:40.793993 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:40.792849    4447 generic.go:356] "Generic (PLEG): container finished" podID="1ef89243-9c18-42f8-92cc-e04cd350a4dc" containerID="d50acf049a18176fd360c81c1e298fba2d38b8f02c18af9d9c1370b40f11e097" exitCode=0
Oct 31 03:23:40.793993 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:40.793003    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5" event={"ID":"1ef89243-9c18-42f8-92cc-e04cd350a4dc","Type":"ContainerDied","Data":"d50acf049a18176fd360c81c1e298fba2d38b8f02c18af9d9c1370b40f11e097"}
Oct 31 03:23:40.799913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:40.797985    4447 generic.go:356] "Generic (PLEG): container finished" podID="206a4f55-822f-4ec0-ac3d-4ff8df04f2dd" containerID="91d1843a49d61e4af66a887a4e41fa083d9ed086c94a6357dcdae022ecde6279" exitCode=0
Oct 31 03:23:40.799913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:40.798086    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk" event={"ID":"206a4f55-822f-4ec0-ac3d-4ff8df04f2dd","Type":"ContainerDied","Data":"91d1843a49d61e4af66a887a4e41fa083d9ed086c94a6357dcdae022ecde6279"}
Oct 31 03:23:44.835932 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:44.835827    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb" event={"ID":"2c3b7b38-a190-451c-bd75-bc48ed07b5e5","Type":"ContainerStarted","Data":"8a80ca8dbb6950ccb6a3aba4cacc1988f856a17e696ca832548666a23188d5ba"}
Oct 31 03:23:45.846006 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:45.845967    4447 generic.go:356] "Generic (PLEG): container finished" podID="2c3b7b38-a190-451c-bd75-bc48ed07b5e5" containerID="8a80ca8dbb6950ccb6a3aba4cacc1988f856a17e696ca832548666a23188d5ba" exitCode=0
Oct 31 03:23:45.846456 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:45.846097    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb" event={"ID":"2c3b7b38-a190-451c-bd75-bc48ed07b5e5","Type":"ContainerDied","Data":"8a80ca8dbb6950ccb6a3aba4cacc1988f856a17e696ca832548666a23188d5ba"}
Oct 31 03:23:46.855656 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:46.855574    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk" event={"ID":"02a07d7a-d488-4f80-8d96-323b749ffdbd","Type":"ContainerStarted","Data":"9d0d03eefe72987662c05009edef55b352161570dc6a6e5ff771c24ce85e7759"}
Oct 31 03:23:46.862328 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:46.861674    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm" event={"ID":"59be87b9-6885-4ab3-8705-580a8ba49625","Type":"ContainerStarted","Data":"9f6f7973c5dfa233e1deb09a3099f27e267ebb05876d106ac8a7c4d06b3dd6a9"}
Oct 31 03:23:46.867239 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:46.867027    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql" event={"ID":"7a682781-0d16-480f-9610-a8bfb1abf481","Type":"ContainerStarted","Data":"59001bbc3eb58de2966b130a4c322ac095aab991e40637fc6384956e44b7f5f6"}
Oct 31 03:23:47.041101 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:47.041070    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb"
Oct 31 03:23:47.065956 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:47.065924    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c3b7b38-a190-451c-bd75-bc48ed07b5e5-kube-api-access-4gk8n\" (UniqueName: \"kubernetes.io/projected/2c3b7b38-a190-451c-bd75-bc48ed07b5e5-kube-api-access-4gk8n\") pod \"2c3b7b38-a190-451c-bd75-bc48ed07b5e5\" (UID: \"2c3b7b38-a190-451c-bd75-bc48ed07b5e5\") "
Oct 31 03:23:47.066116 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:47.065982    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3b7b38-a190-451c-bd75-bc48ed07b5e5-util\" (UniqueName: \"kubernetes.io/empty-dir/2c3b7b38-a190-451c-bd75-bc48ed07b5e5-util\") pod \"2c3b7b38-a190-451c-bd75-bc48ed07b5e5\" (UID: \"2c3b7b38-a190-451c-bd75-bc48ed07b5e5\") "
Oct 31 03:23:47.066116 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:47.066033    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3b7b38-a190-451c-bd75-bc48ed07b5e5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c3b7b38-a190-451c-bd75-bc48ed07b5e5-bundle\") pod \"2c3b7b38-a190-451c-bd75-bc48ed07b5e5\" (UID: \"2c3b7b38-a190-451c-bd75-bc48ed07b5e5\") "
Oct 31 03:23:47.067078 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:47.067052    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3b7b38-a190-451c-bd75-bc48ed07b5e5-bundle" pod "2c3b7b38-a190-451c-bd75-bc48ed07b5e5" (UID: "2c3b7b38-a190-451c-bd75-bc48ed07b5e5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:23:47.070186 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:47.070162    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3b7b38-a190-451c-bd75-bc48ed07b5e5-kube-api-access-4gk8n" pod "2c3b7b38-a190-451c-bd75-bc48ed07b5e5" (UID: "2c3b7b38-a190-451c-bd75-bc48ed07b5e5"). InnerVolumeSpecName "kube-api-access-4gk8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:23:47.073475 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:47.073449    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3b7b38-a190-451c-bd75-bc48ed07b5e5-util" pod "2c3b7b38-a190-451c-bd75-bc48ed07b5e5" (UID: "2c3b7b38-a190-451c-bd75-bc48ed07b5e5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:23:47.118194 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:47.118083    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql" podStartSLOduration=2.605631681 podStartE2EDuration="28.118065505s" totalImagesPullingTime="25.512433824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:47.117462774 +0000 UTC m=+252.628445163" watchObservedRunningTime="2026-10-31 03:23:47.118065505 +0000 UTC m=+252.629047883"
Oct 31 03:23:47.118623 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:47.118582    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk" podStartSLOduration=2.831127155 podStartE2EDuration="24.11857118s" totalImagesPullingTime="21.287444025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:46.99658805 +0000 UTC m=+252.507570438" watchObservedRunningTime="2026-10-31 03:23:47.11857118 +0000 UTC m=+252.629553558"
Oct 31 03:23:47.166849 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:47.166773    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gk8n\" (UniqueName: \"kubernetes.io/projected/2c3b7b38-a190-451c-bd75-bc48ed07b5e5-kube-api-access-4gk8n\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:23:47.166849 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:47.166807    4447 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c3b7b38-a190-451c-bd75-bc48ed07b5e5-util\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:23:47.166849 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:47.166822    4447 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c3b7b38-a190-451c-bd75-bc48ed07b5e5-bundle\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:23:47.239311 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:47.239254    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm" podStartSLOduration=2.633695206 podStartE2EDuration="23.239233973s" totalImagesPullingTime="20.605538767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:47.236582075 +0000 UTC m=+252.747564452" watchObservedRunningTime="2026-10-31 03:23:47.239233973 +0000 UTC m=+252.750216352"
Oct 31 03:23:47.875191 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:47.875153    4447 generic.go:356] "Generic (PLEG): container finished" podID="59be87b9-6885-4ab3-8705-580a8ba49625" containerID="9f6f7973c5dfa233e1deb09a3099f27e267ebb05876d106ac8a7c4d06b3dd6a9" exitCode=0
Oct 31 03:23:47.875649 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:47.875245    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm" event={"ID":"59be87b9-6885-4ab3-8705-580a8ba49625","Type":"ContainerDied","Data":"9f6f7973c5dfa233e1deb09a3099f27e267ebb05876d106ac8a7c4d06b3dd6a9"}
Oct 31 03:23:47.877746 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:47.877718    4447 generic.go:356] "Generic (PLEG): container finished" podID="7a682781-0d16-480f-9610-a8bfb1abf481" containerID="59001bbc3eb58de2966b130a4c322ac095aab991e40637fc6384956e44b7f5f6" exitCode=0
Oct 31 03:23:47.877902 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:47.877799    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql" event={"ID":"7a682781-0d16-480f-9610-a8bfb1abf481","Type":"ContainerDied","Data":"59001bbc3eb58de2966b130a4c322ac095aab991e40637fc6384956e44b7f5f6"}
Oct 31 03:23:47.880278 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:47.880237    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj" event={"ID":"e2298be1-df21-4e9b-b1ef-75559fb24494","Type":"ContainerStarted","Data":"7399d17e0fa212c59589def8736936934eb2ccc565b1342ee1da1aeec18b8c61"}
Oct 31 03:23:47.883655 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:47.883623    4447 generic.go:356] "Generic (PLEG): container finished" podID="02a07d7a-d488-4f80-8d96-323b749ffdbd" containerID="9d0d03eefe72987662c05009edef55b352161570dc6a6e5ff771c24ce85e7759" exitCode=0
Oct 31 03:23:47.883765 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:47.883731    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk" event={"ID":"02a07d7a-d488-4f80-8d96-323b749ffdbd","Type":"ContainerDied","Data":"9d0d03eefe72987662c05009edef55b352161570dc6a6e5ff771c24ce85e7759"}
Oct 31 03:23:47.885851 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:47.885825    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb" event={"ID":"2c3b7b38-a190-451c-bd75-bc48ed07b5e5","Type":"ContainerDied","Data":"041d61a721642787286aec6a3604b181d489267212f4bb5a1c42b4b52376a87c"}
Oct 31 03:23:47.885851 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:47.885848    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71x6wmb"
Oct 31 03:23:47.886039 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:47.885867    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="041d61a721642787286aec6a3604b181d489267212f4bb5a1c42b4b52376a87c"
Oct 31 03:23:48.118046 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:48.117987    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj" podStartSLOduration=2.921763292 podStartE2EDuration="24.117967109s" totalImagesPullingTime="21.196203817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:48.117341608 +0000 UTC m=+253.628323987" watchObservedRunningTime="2026-10-31 03:23:48.117967109 +0000 UTC m=+253.628949487"
Oct 31 03:23:48.898338 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:48.898303    4447 generic.go:356] "Generic (PLEG): container finished" podID="e2298be1-df21-4e9b-b1ef-75559fb24494" containerID="7399d17e0fa212c59589def8736936934eb2ccc565b1342ee1da1aeec18b8c61" exitCode=0
Oct 31 03:23:48.898716 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:48.898382    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj" event={"ID":"e2298be1-df21-4e9b-b1ef-75559fb24494","Type":"ContainerDied","Data":"7399d17e0fa212c59589def8736936934eb2ccc565b1342ee1da1aeec18b8c61"}
Oct 31 03:23:48.900543 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:48.900520    4447 generic.go:356] "Generic (PLEG): container finished" podID="1ef89243-9c18-42f8-92cc-e04cd350a4dc" containerID="f53d0666b80f64fc990f36b1a0d859b31a95ad10ed0559494310137a6a861f87" exitCode=0
Oct 31 03:23:48.900657 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:48.900603    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5" event={"ID":"1ef89243-9c18-42f8-92cc-e04cd350a4dc","Type":"ContainerDied","Data":"f53d0666b80f64fc990f36b1a0d859b31a95ad10ed0559494310137a6a861f87"}
Oct 31 03:23:48.902548 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:48.902528    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk" event={"ID":"206a4f55-822f-4ec0-ac3d-4ff8df04f2dd","Type":"ContainerStarted","Data":"b1a08e324aa0435379e4004e34e3ca0b1650c3a348746c8138ef711796a91bf9"}
Oct 31 03:23:49.013922 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:23:49.013770    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:23:49.083915 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.083868    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk"
Oct 31 03:23:49.086937 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.086918    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm"
Oct 31 03:23:49.090260 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.090244    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql"
Oct 31 03:23:49.186049 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.185968    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59be87b9-6885-4ab3-8705-580a8ba49625-bundle\" (UniqueName: \"kubernetes.io/empty-dir/59be87b9-6885-4ab3-8705-580a8ba49625-bundle\") pod \"59be87b9-6885-4ab3-8705-580a8ba49625\" (UID: \"59be87b9-6885-4ab3-8705-580a8ba49625\") "
Oct 31 03:23:49.186049 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.186021    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a682781-0d16-480f-9610-a8bfb1abf481-util\" (UniqueName: \"kubernetes.io/empty-dir/7a682781-0d16-480f-9610-a8bfb1abf481-util\") pod \"7a682781-0d16-480f-9610-a8bfb1abf481\" (UID: \"7a682781-0d16-480f-9610-a8bfb1abf481\") "
Oct 31 03:23:49.186049 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.186051    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59be87b9-6885-4ab3-8705-580a8ba49625-kube-api-access-jjsv8\" (UniqueName: \"kubernetes.io/projected/59be87b9-6885-4ab3-8705-580a8ba49625-kube-api-access-jjsv8\") pod \"59be87b9-6885-4ab3-8705-580a8ba49625\" (UID: \"59be87b9-6885-4ab3-8705-580a8ba49625\") "
Oct 31 03:23:49.186305 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.186115    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02a07d7a-d488-4f80-8d96-323b749ffdbd-kube-api-access-2v4xm\" (UniqueName: \"kubernetes.io/projected/02a07d7a-d488-4f80-8d96-323b749ffdbd-kube-api-access-2v4xm\") pod \"02a07d7a-d488-4f80-8d96-323b749ffdbd\" (UID: \"02a07d7a-d488-4f80-8d96-323b749ffdbd\") "
Oct 31 03:23:49.186305 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.186158    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a682781-0d16-480f-9610-a8bfb1abf481-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a682781-0d16-480f-9610-a8bfb1abf481-bundle\") pod \"7a682781-0d16-480f-9610-a8bfb1abf481\" (UID: \"7a682781-0d16-480f-9610-a8bfb1abf481\") "
Oct 31 03:23:49.186305 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.186185    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a682781-0d16-480f-9610-a8bfb1abf481-kube-api-access-5bqb2\" (UniqueName: \"kubernetes.io/projected/7a682781-0d16-480f-9610-a8bfb1abf481-kube-api-access-5bqb2\") pod \"7a682781-0d16-480f-9610-a8bfb1abf481\" (UID: \"7a682781-0d16-480f-9610-a8bfb1abf481\") "
Oct 31 03:23:49.186305 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.186215    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a07d7a-d488-4f80-8d96-323b749ffdbd-util\" (UniqueName: \"kubernetes.io/empty-dir/02a07d7a-d488-4f80-8d96-323b749ffdbd-util\") pod \"02a07d7a-d488-4f80-8d96-323b749ffdbd\" (UID: \"02a07d7a-d488-4f80-8d96-323b749ffdbd\") "
Oct 31 03:23:49.186305 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.186251    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a07d7a-d488-4f80-8d96-323b749ffdbd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/02a07d7a-d488-4f80-8d96-323b749ffdbd-bundle\") pod \"02a07d7a-d488-4f80-8d96-323b749ffdbd\" (UID: \"02a07d7a-d488-4f80-8d96-323b749ffdbd\") "
Oct 31 03:23:49.186671 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.186640    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59be87b9-6885-4ab3-8705-580a8ba49625-bundle" pod "59be87b9-6885-4ab3-8705-580a8ba49625" (UID: "59be87b9-6885-4ab3-8705-580a8ba49625"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:23:49.186778 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.186695    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59be87b9-6885-4ab3-8705-580a8ba49625-util\" (UniqueName: \"kubernetes.io/empty-dir/59be87b9-6885-4ab3-8705-580a8ba49625-util\") pod \"59be87b9-6885-4ab3-8705-580a8ba49625\" (UID: \"59be87b9-6885-4ab3-8705-580a8ba49625\") "
Oct 31 03:23:49.187391 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.187003    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a682781-0d16-480f-9610-a8bfb1abf481-bundle" pod "7a682781-0d16-480f-9610-a8bfb1abf481" (UID: "7a682781-0d16-480f-9610-a8bfb1abf481"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:23:49.187391 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.187336    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a07d7a-d488-4f80-8d96-323b749ffdbd-bundle" pod "02a07d7a-d488-4f80-8d96-323b749ffdbd" (UID: "02a07d7a-d488-4f80-8d96-323b749ffdbd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:23:49.187656 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.187633    4447 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59be87b9-6885-4ab3-8705-580a8ba49625-bundle\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:23:49.187742 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.187662    4447 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a682781-0d16-480f-9610-a8bfb1abf481-bundle\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:23:49.188620 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.188597    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59be87b9-6885-4ab3-8705-580a8ba49625-kube-api-access-jjsv8" pod "59be87b9-6885-4ab3-8705-580a8ba49625" (UID: "59be87b9-6885-4ab3-8705-580a8ba49625"). InnerVolumeSpecName "kube-api-access-jjsv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:23:49.188728 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.188682    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a682781-0d16-480f-9610-a8bfb1abf481-kube-api-access-5bqb2" pod "7a682781-0d16-480f-9610-a8bfb1abf481" (UID: "7a682781-0d16-480f-9610-a8bfb1abf481"). InnerVolumeSpecName "kube-api-access-5bqb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:23:49.188808 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.188788    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a07d7a-d488-4f80-8d96-323b749ffdbd-kube-api-access-2v4xm" pod "02a07d7a-d488-4f80-8d96-323b749ffdbd" (UID: "02a07d7a-d488-4f80-8d96-323b749ffdbd"). InnerVolumeSpecName "kube-api-access-2v4xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:23:49.190773 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.190754    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a682781-0d16-480f-9610-a8bfb1abf481-util" pod "7a682781-0d16-480f-9610-a8bfb1abf481" (UID: "7a682781-0d16-480f-9610-a8bfb1abf481"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:23:49.193546 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.193523    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59be87b9-6885-4ab3-8705-580a8ba49625-util" pod "59be87b9-6885-4ab3-8705-580a8ba49625" (UID: "59be87b9-6885-4ab3-8705-580a8ba49625"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:23:49.193754 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.193732    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a07d7a-d488-4f80-8d96-323b749ffdbd-util" pod "02a07d7a-d488-4f80-8d96-323b749ffdbd" (UID: "02a07d7a-d488-4f80-8d96-323b749ffdbd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:23:49.288816 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.288784    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bqb2\" (UniqueName: \"kubernetes.io/projected/7a682781-0d16-480f-9610-a8bfb1abf481-kube-api-access-5bqb2\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:23:49.288816 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.288812    4447 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02a07d7a-d488-4f80-8d96-323b749ffdbd-util\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:23:49.288816 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.288822    4447 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02a07d7a-d488-4f80-8d96-323b749ffdbd-bundle\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:23:49.289052 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.288831    4447 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59be87b9-6885-4ab3-8705-580a8ba49625-util\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:23:49.289052 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.288839    4447 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a682781-0d16-480f-9610-a8bfb1abf481-util\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:23:49.289052 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.288846    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjsv8\" (UniqueName: \"kubernetes.io/projected/59be87b9-6885-4ab3-8705-580a8ba49625-kube-api-access-jjsv8\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:23:49.289052 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.288854    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v4xm\" (UniqueName: \"kubernetes.io/projected/02a07d7a-d488-4f80-8d96-323b749ffdbd-kube-api-access-2v4xm\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:23:49.909264 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.909229    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm"
Oct 31 03:23:49.909654 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.909226    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnqbwm" event={"ID":"59be87b9-6885-4ab3-8705-580a8ba49625","Type":"ContainerDied","Data":"11887a149e80b9fa5342da9a001057300c3d2e0a78e12943b91baaea11741597"}
Oct 31 03:23:49.909654 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.909332    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11887a149e80b9fa5342da9a001057300c3d2e0a78e12943b91baaea11741597"
Oct 31 03:23:49.911231 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.911208    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql"
Oct 31 03:23:49.911231 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.911222    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2gvql" event={"ID":"7a682781-0d16-480f-9610-a8bfb1abf481","Type":"ContainerDied","Data":"21c409b7338f0157e81c538007c2b22fe13de4da5938d1bc4a82d2fdd7f08182"}
Oct 31 03:23:49.911436 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.911259    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21c409b7338f0157e81c538007c2b22fe13de4da5938d1bc4a82d2fdd7f08182"
Oct 31 03:23:49.913253 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.913236    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk"
Oct 31 03:23:49.913367 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.913266    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5jsk" event={"ID":"02a07d7a-d488-4f80-8d96-323b749ffdbd","Type":"ContainerDied","Data":"f83655603f468b535f68b6e676d8d4fcc9256de2a5b6e061ffeb437b974f44d6"}
Oct 31 03:23:49.913367 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.913298    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f83655603f468b535f68b6e676d8d4fcc9256de2a5b6e061ffeb437b974f44d6"
Oct 31 03:23:49.915801 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.915773    4447 generic.go:356] "Generic (PLEG): container finished" podID="206a4f55-822f-4ec0-ac3d-4ff8df04f2dd" containerID="b1a08e324aa0435379e4004e34e3ca0b1650c3a348746c8138ef711796a91bf9" exitCode=0
Oct 31 03:23:49.915935 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:49.915825    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk" event={"ID":"206a4f55-822f-4ec0-ac3d-4ff8df04f2dd","Type":"ContainerDied","Data":"b1a08e324aa0435379e4004e34e3ca0b1650c3a348746c8138ef711796a91bf9"}
Oct 31 03:23:50.061960 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.061936    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk"
Oct 31 03:23:50.077727 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.077705    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"]
Oct 31 03:23:50.078255 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078232    4447 cpu_manager.go:414] "containerMap: removing container" podUID="7a682781-0d16-480f-9610-a8bfb1abf481" containerName="util"
Oct 31 03:23:50.078255 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078250    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a682781-0d16-480f-9610-a8bfb1abf481" containerName="util"
Oct 31 03:23:50.078357 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078271    4447 cpu_manager.go:414] "containerMap: removing container" podUID="7a682781-0d16-480f-9610-a8bfb1abf481" containerName="pull"
Oct 31 03:23:50.078357 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078279    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a682781-0d16-480f-9610-a8bfb1abf481" containerName="pull"
Oct 31 03:23:50.078357 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078296    4447 cpu_manager.go:414] "containerMap: removing container" podUID="2c3b7b38-a190-451c-bd75-bc48ed07b5e5" containerName="util"
Oct 31 03:23:50.078357 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078303    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c3b7b38-a190-451c-bd75-bc48ed07b5e5" containerName="util"
Oct 31 03:23:50.078357 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078317    4447 cpu_manager.go:414] "containerMap: removing container" podUID="59be87b9-6885-4ab3-8705-580a8ba49625" containerName="util"
Oct 31 03:23:50.078357 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078323    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59be87b9-6885-4ab3-8705-580a8ba49625" containerName="util"
Oct 31 03:23:50.078357 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078334    4447 cpu_manager.go:414] "containerMap: removing container" podUID="59be87b9-6885-4ab3-8705-580a8ba49625" containerName="pull"
Oct 31 03:23:50.078357 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078340    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59be87b9-6885-4ab3-8705-580a8ba49625" containerName="pull"
Oct 31 03:23:50.078357 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078352    4447 cpu_manager.go:414] "containerMap: removing container" podUID="206a4f55-822f-4ec0-ac3d-4ff8df04f2dd" containerName="util"
Oct 31 03:23:50.078627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078360    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="206a4f55-822f-4ec0-ac3d-4ff8df04f2dd" containerName="util"
Oct 31 03:23:50.078627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078370    4447 cpu_manager.go:414] "containerMap: removing container" podUID="206a4f55-822f-4ec0-ac3d-4ff8df04f2dd" containerName="pull"
Oct 31 03:23:50.078627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078376    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="206a4f55-822f-4ec0-ac3d-4ff8df04f2dd" containerName="pull"
Oct 31 03:23:50.078627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078384    4447 cpu_manager.go:414] "containerMap: removing container" podUID="2c3b7b38-a190-451c-bd75-bc48ed07b5e5" containerName="extract"
Oct 31 03:23:50.078627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078391    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c3b7b38-a190-451c-bd75-bc48ed07b5e5" containerName="extract"
Oct 31 03:23:50.078627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078403    4447 cpu_manager.go:414] "containerMap: removing container" podUID="02a07d7a-d488-4f80-8d96-323b749ffdbd" containerName="util"
Oct 31 03:23:50.078627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078411    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02a07d7a-d488-4f80-8d96-323b749ffdbd" containerName="util"
Oct 31 03:23:50.078627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078425    4447 cpu_manager.go:414] "containerMap: removing container" podUID="2c3b7b38-a190-451c-bd75-bc48ed07b5e5" containerName="pull"
Oct 31 03:23:50.078627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078432    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c3b7b38-a190-451c-bd75-bc48ed07b5e5" containerName="pull"
Oct 31 03:23:50.078627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078447    4447 cpu_manager.go:414] "containerMap: removing container" podUID="59be87b9-6885-4ab3-8705-580a8ba49625" containerName="extract"
Oct 31 03:23:50.078627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078453    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59be87b9-6885-4ab3-8705-580a8ba49625" containerName="extract"
Oct 31 03:23:50.078627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078463    4447 cpu_manager.go:414] "containerMap: removing container" podUID="206a4f55-822f-4ec0-ac3d-4ff8df04f2dd" containerName="extract"
Oct 31 03:23:50.078627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078469    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="206a4f55-822f-4ec0-ac3d-4ff8df04f2dd" containerName="extract"
Oct 31 03:23:50.078627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078489    4447 cpu_manager.go:414] "containerMap: removing container" podUID="7a682781-0d16-480f-9610-a8bfb1abf481" containerName="extract"
Oct 31 03:23:50.078627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078495    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a682781-0d16-480f-9610-a8bfb1abf481" containerName="extract"
Oct 31 03:23:50.078627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078508    4447 cpu_manager.go:414] "containerMap: removing container" podUID="02a07d7a-d488-4f80-8d96-323b749ffdbd" containerName="extract"
Oct 31 03:23:50.078627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078515    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02a07d7a-d488-4f80-8d96-323b749ffdbd" containerName="extract"
Oct 31 03:23:50.078627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078523    4447 cpu_manager.go:414] "containerMap: removing container" podUID="02a07d7a-d488-4f80-8d96-323b749ffdbd" containerName="pull"
Oct 31 03:23:50.078627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078529    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02a07d7a-d488-4f80-8d96-323b749ffdbd" containerName="pull"
Oct 31 03:23:50.078627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078605    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c3b7b38-a190-451c-bd75-bc48ed07b5e5" containerName="extract"
Oct 31 03:23:50.078627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078617    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="59be87b9-6885-4ab3-8705-580a8ba49625" containerName="extract"
Oct 31 03:23:50.078627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078626    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="206a4f55-822f-4ec0-ac3d-4ff8df04f2dd" containerName="extract"
Oct 31 03:23:50.078627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078636    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a682781-0d16-480f-9610-a8bfb1abf481" containerName="extract"
Oct 31 03:23:50.080040 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.078645    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="02a07d7a-d488-4f80-8d96-323b749ffdbd" containerName="extract"
Oct 31 03:23:50.082295 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.082275    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"
Oct 31 03:23:50.094840 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.094822    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/206a4f55-822f-4ec0-ac3d-4ff8df04f2dd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/206a4f55-822f-4ec0-ac3d-4ff8df04f2dd-bundle\") pod \"206a4f55-822f-4ec0-ac3d-4ff8df04f2dd\" (UID: \"206a4f55-822f-4ec0-ac3d-4ff8df04f2dd\") "
Oct 31 03:23:50.094963 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.094923    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/206a4f55-822f-4ec0-ac3d-4ff8df04f2dd-kube-api-access-4pf9l\" (UniqueName: \"kubernetes.io/projected/206a4f55-822f-4ec0-ac3d-4ff8df04f2dd-kube-api-access-4pf9l\") pod \"206a4f55-822f-4ec0-ac3d-4ff8df04f2dd\" (UID: \"206a4f55-822f-4ec0-ac3d-4ff8df04f2dd\") "
Oct 31 03:23:50.095048 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.095031    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/206a4f55-822f-4ec0-ac3d-4ff8df04f2dd-util\" (UniqueName: \"kubernetes.io/empty-dir/206a4f55-822f-4ec0-ac3d-4ff8df04f2dd-util\") pod \"206a4f55-822f-4ec0-ac3d-4ff8df04f2dd\" (UID: \"206a4f55-822f-4ec0-ac3d-4ff8df04f2dd\") "
Oct 31 03:23:50.095180 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.095160    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxpcp\" (UniqueName: \"kubernetes.io/projected/518628c8-8995-4344-b591-ed5afc23d78b-kube-api-access-xxpcp\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-shc8p\" (UID: \"518628c8-8995-4344-b591-ed5afc23d78b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"
Oct 31 03:23:50.095265 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.095251    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/518628c8-8995-4344-b591-ed5afc23d78b-metrics-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-shc8p\" (UID: \"518628c8-8995-4344-b591-ed5afc23d78b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"
Oct 31 03:23:50.095381 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.095360    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/518628c8-8995-4344-b591-ed5afc23d78b-tmp\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-shc8p\" (UID: \"518628c8-8995-4344-b591-ed5afc23d78b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"
Oct 31 03:23:50.095484 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.095423    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/518628c8-8995-4344-b591-ed5afc23d78b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-shc8p\" (UID: \"518628c8-8995-4344-b591-ed5afc23d78b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"
Oct 31 03:23:50.095546 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.095502    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206a4f55-822f-4ec0-ac3d-4ff8df04f2dd-bundle" pod "206a4f55-822f-4ec0-ac3d-4ff8df04f2dd" (UID: "206a4f55-822f-4ec0-ac3d-4ff8df04f2dd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:23:50.095546 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.095504    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/518628c8-8995-4344-b591-ed5afc23d78b-webhook-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-shc8p\" (UID: \"518628c8-8995-4344-b591-ed5afc23d78b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"
Oct 31 03:23:50.095647 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.095550    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/518628c8-8995-4344-b591-ed5afc23d78b-manifests\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-shc8p\" (UID: \"518628c8-8995-4344-b591-ed5afc23d78b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"
Oct 31 03:23:50.095647 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.095600    4447 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/206a4f55-822f-4ec0-ac3d-4ff8df04f2dd-bundle\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:23:50.098656 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.098636    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/206a4f55-822f-4ec0-ac3d-4ff8df04f2dd-kube-api-access-4pf9l" pod "206a4f55-822f-4ec0-ac3d-4ff8df04f2dd" (UID: "206a4f55-822f-4ec0-ac3d-4ff8df04f2dd"). InnerVolumeSpecName "kube-api-access-4pf9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:23:50.098820 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.098807    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj"
Oct 31 03:23:50.102823 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.102794    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/206a4f55-822f-4ec0-ac3d-4ff8df04f2dd-util" pod "206a4f55-822f-4ec0-ac3d-4ff8df04f2dd" (UID: "206a4f55-822f-4ec0-ac3d-4ff8df04f2dd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:23:50.114397 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.114378    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5"
Oct 31 03:23:50.144750 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.144675    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 03:23:50.144918 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.144774    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 03:23:50.144918 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.144808    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 03:23:50.144918 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.144825    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 03:23:50.145042 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.145014    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 03:23:50.145077 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.145042    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jccsh\""
Oct 31 03:23:50.196349 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.196322    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef89243-9c18-42f8-92cc-e04cd350a4dc-kube-api-access-p4scw\" (UniqueName: \"kubernetes.io/projected/1ef89243-9c18-42f8-92cc-e04cd350a4dc-kube-api-access-p4scw\") pod \"1ef89243-9c18-42f8-92cc-e04cd350a4dc\" (UID: \"1ef89243-9c18-42f8-92cc-e04cd350a4dc\") "
Oct 31 03:23:50.196547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.196396    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef89243-9c18-42f8-92cc-e04cd350a4dc-util\" (UniqueName: \"kubernetes.io/empty-dir/1ef89243-9c18-42f8-92cc-e04cd350a4dc-util\") pod \"1ef89243-9c18-42f8-92cc-e04cd350a4dc\" (UID: \"1ef89243-9c18-42f8-92cc-e04cd350a4dc\") "
Oct 31 03:23:50.196547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.196425    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef89243-9c18-42f8-92cc-e04cd350a4dc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ef89243-9c18-42f8-92cc-e04cd350a4dc-bundle\") pod \"1ef89243-9c18-42f8-92cc-e04cd350a4dc\" (UID: \"1ef89243-9c18-42f8-92cc-e04cd350a4dc\") "
Oct 31 03:23:50.196547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.196466    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2298be1-df21-4e9b-b1ef-75559fb24494-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2298be1-df21-4e9b-b1ef-75559fb24494-bundle\") pod \"e2298be1-df21-4e9b-b1ef-75559fb24494\" (UID: \"e2298be1-df21-4e9b-b1ef-75559fb24494\") "
Oct 31 03:23:50.196547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.196484    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2298be1-df21-4e9b-b1ef-75559fb24494-kube-api-access-vpmv6\" (UniqueName: \"kubernetes.io/projected/e2298be1-df21-4e9b-b1ef-75559fb24494-kube-api-access-vpmv6\") pod \"e2298be1-df21-4e9b-b1ef-75559fb24494\" (UID: \"e2298be1-df21-4e9b-b1ef-75559fb24494\") "
Oct 31 03:23:50.196547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.196503    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2298be1-df21-4e9b-b1ef-75559fb24494-util\" (UniqueName: \"kubernetes.io/empty-dir/e2298be1-df21-4e9b-b1ef-75559fb24494-util\") pod \"e2298be1-df21-4e9b-b1ef-75559fb24494\" (UID: \"e2298be1-df21-4e9b-b1ef-75559fb24494\") "
Oct 31 03:23:50.196802 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.196569    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/518628c8-8995-4344-b591-ed5afc23d78b-manifests\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-shc8p\" (UID: \"518628c8-8995-4344-b591-ed5afc23d78b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"
Oct 31 03:23:50.196802 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.196610    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxpcp\" (UniqueName: \"kubernetes.io/projected/518628c8-8995-4344-b591-ed5afc23d78b-kube-api-access-xxpcp\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-shc8p\" (UID: \"518628c8-8995-4344-b591-ed5afc23d78b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"
Oct 31 03:23:50.196802 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.196689    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/518628c8-8995-4344-b591-ed5afc23d78b-metrics-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-shc8p\" (UID: \"518628c8-8995-4344-b591-ed5afc23d78b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"
Oct 31 03:23:50.196802 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.196734    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/518628c8-8995-4344-b591-ed5afc23d78b-tmp\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-shc8p\" (UID: \"518628c8-8995-4344-b591-ed5afc23d78b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"
Oct 31 03:23:50.196802 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.196797    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/518628c8-8995-4344-b591-ed5afc23d78b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-shc8p\" (UID: \"518628c8-8995-4344-b591-ed5afc23d78b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"
Oct 31 03:23:50.197085 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.196819    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/518628c8-8995-4344-b591-ed5afc23d78b-webhook-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-shc8p\" (UID: \"518628c8-8995-4344-b591-ed5afc23d78b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"
Oct 31 03:23:50.197085 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.196856    4447 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/206a4f55-822f-4ec0-ac3d-4ff8df04f2dd-util\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:23:50.197085 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.196867    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pf9l\" (UniqueName: \"kubernetes.io/projected/206a4f55-822f-4ec0-ac3d-4ff8df04f2dd-kube-api-access-4pf9l\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:23:50.197085 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.197037    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef89243-9c18-42f8-92cc-e04cd350a4dc-bundle" pod "1ef89243-9c18-42f8-92cc-e04cd350a4dc" (UID: "1ef89243-9c18-42f8-92cc-e04cd350a4dc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:23:50.197268 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.197133    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/518628c8-8995-4344-b591-ed5afc23d78b-manifests\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-shc8p\" (UID: \"518628c8-8995-4344-b591-ed5afc23d78b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"
Oct 31 03:23:50.197599 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.197569    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2298be1-df21-4e9b-b1ef-75559fb24494-bundle" pod "e2298be1-df21-4e9b-b1ef-75559fb24494" (UID: "e2298be1-df21-4e9b-b1ef-75559fb24494"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:23:50.199761 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.199717    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2298be1-df21-4e9b-b1ef-75559fb24494-kube-api-access-vpmv6" pod "e2298be1-df21-4e9b-b1ef-75559fb24494" (UID: "e2298be1-df21-4e9b-b1ef-75559fb24494"). InnerVolumeSpecName "kube-api-access-vpmv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:23:50.199926 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.199771    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/518628c8-8995-4344-b591-ed5afc23d78b-webhook-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-shc8p\" (UID: \"518628c8-8995-4344-b591-ed5afc23d78b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"
Oct 31 03:23:50.200120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.200095    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef89243-9c18-42f8-92cc-e04cd350a4dc-kube-api-access-p4scw" pod "1ef89243-9c18-42f8-92cc-e04cd350a4dc" (UID: "1ef89243-9c18-42f8-92cc-e04cd350a4dc"). InnerVolumeSpecName "kube-api-access-p4scw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:23:50.200265 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.200231    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/518628c8-8995-4344-b591-ed5afc23d78b-tmp\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-shc8p\" (UID: \"518628c8-8995-4344-b591-ed5afc23d78b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"
Oct 31 03:23:50.200326 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.200284    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/518628c8-8995-4344-b591-ed5afc23d78b-metrics-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-shc8p\" (UID: \"518628c8-8995-4344-b591-ed5afc23d78b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"
Oct 31 03:23:50.200493 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.200478    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/518628c8-8995-4344-b591-ed5afc23d78b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-shc8p\" (UID: \"518628c8-8995-4344-b591-ed5afc23d78b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"
Oct 31 03:23:50.203614 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.203595    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2298be1-df21-4e9b-b1ef-75559fb24494-util" pod "e2298be1-df21-4e9b-b1ef-75559fb24494" (UID: "e2298be1-df21-4e9b-b1ef-75559fb24494"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:23:50.203731 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.203710    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef89243-9c18-42f8-92cc-e04cd350a4dc-util" pod "1ef89243-9c18-42f8-92cc-e04cd350a4dc" (UID: "1ef89243-9c18-42f8-92cc-e04cd350a4dc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:23:50.212158 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.212139    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"]
Oct 31 03:23:50.265074 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.265040    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxpcp\" (UniqueName: \"kubernetes.io/projected/518628c8-8995-4344-b591-ed5afc23d78b-kube-api-access-xxpcp\") pod \"opendatahub-operator-controller-manager-84dfd97fcd-shc8p\" (UID: \"518628c8-8995-4344-b591-ed5afc23d78b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"
Oct 31 03:23:50.297964 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.297935    4447 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2298be1-df21-4e9b-b1ef-75559fb24494-bundle\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:23:50.297964 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.297957    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpmv6\" (UniqueName: \"kubernetes.io/projected/e2298be1-df21-4e9b-b1ef-75559fb24494-kube-api-access-vpmv6\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:23:50.297964 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.297967    4447 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2298be1-df21-4e9b-b1ef-75559fb24494-util\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:23:50.297964 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.297976    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4scw\" (UniqueName: \"kubernetes.io/projected/1ef89243-9c18-42f8-92cc-e04cd350a4dc-kube-api-access-p4scw\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:23:50.298308 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.297985    4447 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ef89243-9c18-42f8-92cc-e04cd350a4dc-util\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:23:50.298308 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.297993    4447 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ef89243-9c18-42f8-92cc-e04cd350a4dc-bundle\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:23:50.411935 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.411831    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"
Oct 31 03:23:50.922182 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.922150    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj"
Oct 31 03:23:50.922182 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.922167    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awr9rj" event={"ID":"e2298be1-df21-4e9b-b1ef-75559fb24494","Type":"ContainerDied","Data":"ed61e90488b3827dfeb112aeeee19df04a133c923df7d2b90404f47ff9db1878"}
Oct 31 03:23:50.922643 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.922218    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed61e90488b3827dfeb112aeeee19df04a133c923df7d2b90404f47ff9db1878"
Oct 31 03:23:50.924131 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.924103    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5" event={"ID":"1ef89243-9c18-42f8-92cc-e04cd350a4dc","Type":"ContainerDied","Data":"3d71978f35ef8f7a99e6708daab98b4ae0c8fb89005027e3782747d1a4dc8e17"}
Oct 31 03:23:50.924262 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.924143    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d71978f35ef8f7a99e6708daab98b4ae0c8fb89005027e3782747d1a4dc8e17"
Oct 31 03:23:50.924262 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.924110    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d782cr5"
Oct 31 03:23:50.926231 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.926210    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk" event={"ID":"206a4f55-822f-4ec0-ac3d-4ff8df04f2dd","Type":"ContainerDied","Data":"39e807dc5036b0b33a2722477a653fb9bd35a435f29480441347f99f5921ccbc"}
Oct 31 03:23:50.926310 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.926244    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39e807dc5036b0b33a2722477a653fb9bd35a435f29480441347f99f5921ccbc"
Oct 31 03:23:50.926533 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.926522    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sqhwk"
Oct 31 03:23:50.938379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:50.938358    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"]
Oct 31 03:23:50.955805 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:23:50.955774    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod518628c8_8995_4344_b591_ed5afc23d78b.slice/crio-1f6d28656252b7b7e25fb0f776ecf4a8f2c9110c038daaccd551798b7ab883a1 WatchSource:0}: Error finding container 1f6d28656252b7b7e25fb0f776ecf4a8f2c9110c038daaccd551798b7ab883a1: Status 404 returned error can't find the container with id 1f6d28656252b7b7e25fb0f776ecf4a8f2c9110c038daaccd551798b7ab883a1
Oct 31 03:23:51.931919 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:51.931864    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p" event={"ID":"518628c8-8995-4344-b591-ed5afc23d78b","Type":"ContainerStarted","Data":"1f6d28656252b7b7e25fb0f776ecf4a8f2c9110c038daaccd551798b7ab883a1"}
Oct 31 03:23:55.953648 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:55.953623    4447 generic.go:356] "Generic (PLEG): container finished" podID="518628c8-8995-4344-b591-ed5afc23d78b" containerID="c1fc4faa273709237b274313e98bd87f96061a2b18b3be107f84bda672ea2f61" exitCode=0
Oct 31 03:23:55.954090 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:55.953708    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p" event={"ID":"518628c8-8995-4344-b591-ed5afc23d78b","Type":"ContainerDied","Data":"c1fc4faa273709237b274313e98bd87f96061a2b18b3be107f84bda672ea2f61"}
Oct 31 03:23:56.959871 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:56.959836    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p" event={"ID":"518628c8-8995-4344-b591-ed5afc23d78b","Type":"ContainerStarted","Data":"49d54cf2d6df8a4feaf3b60a72539f9bf7cf6787c9e52bbeaf43ebf25e4eb5b4"}
Oct 31 03:23:56.960313 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:56.960031    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"
Oct 31 03:23:57.105735 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:57.105626    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p" podStartSLOduration=2.920300766 podStartE2EDuration="8.105611649s" totalImagesPullingTime="4.741273791s" totalInitContainerRuntime="444.037092ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:57.103389746 +0000 UTC m=+262.614372126" watchObservedRunningTime="2026-10-31 03:23:57.105611649 +0000 UTC m=+262.616594026"
Oct 31 03:23:58.858167 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:58.858137    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tp7s8"]
Oct 31 03:23:58.858547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:58.858468    4447 cpu_manager.go:414] "containerMap: removing container" podUID="e2298be1-df21-4e9b-b1ef-75559fb24494" containerName="pull"
Oct 31 03:23:58.858547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:58.858478    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2298be1-df21-4e9b-b1ef-75559fb24494" containerName="pull"
Oct 31 03:23:58.858547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:58.858491    4447 cpu_manager.go:414] "containerMap: removing container" podUID="e2298be1-df21-4e9b-b1ef-75559fb24494" containerName="util"
Oct 31 03:23:58.858547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:58.858496    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2298be1-df21-4e9b-b1ef-75559fb24494" containerName="util"
Oct 31 03:23:58.858547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:58.858511    4447 cpu_manager.go:414] "containerMap: removing container" podUID="1ef89243-9c18-42f8-92cc-e04cd350a4dc" containerName="pull"
Oct 31 03:23:58.858547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:58.858516    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef89243-9c18-42f8-92cc-e04cd350a4dc" containerName="pull"
Oct 31 03:23:58.858547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:58.858525    4447 cpu_manager.go:414] "containerMap: removing container" podUID="e2298be1-df21-4e9b-b1ef-75559fb24494" containerName="extract"
Oct 31 03:23:58.858547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:58.858529    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2298be1-df21-4e9b-b1ef-75559fb24494" containerName="extract"
Oct 31 03:23:58.858547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:58.858542    4447 cpu_manager.go:414] "containerMap: removing container" podUID="1ef89243-9c18-42f8-92cc-e04cd350a4dc" containerName="extract"
Oct 31 03:23:58.858547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:58.858546    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef89243-9c18-42f8-92cc-e04cd350a4dc" containerName="extract"
Oct 31 03:23:58.858834 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:58.858554    4447 cpu_manager.go:414] "containerMap: removing container" podUID="1ef89243-9c18-42f8-92cc-e04cd350a4dc" containerName="util"
Oct 31 03:23:58.858834 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:58.858558    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef89243-9c18-42f8-92cc-e04cd350a4dc" containerName="util"
Oct 31 03:23:58.858834 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:58.858608    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2298be1-df21-4e9b-b1ef-75559fb24494" containerName="extract"
Oct 31 03:23:58.858834 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:58.858615    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef89243-9c18-42f8-92cc-e04cd350a4dc" containerName="extract"
Oct 31 03:23:58.860349 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:58.860335    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tp7s8"
Oct 31 03:23:58.916265 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:58.916157    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-n9qkp\""
Oct 31 03:23:58.965838 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:58.965806    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 31 03:23:58.965838 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:58.965806    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:23:58.968442 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:58.968420    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tp7s8"]
Oct 31 03:23:58.973771 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:58.973751    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqrmz\" (UniqueName: \"kubernetes.io/projected/059ef715-b1d1-47e5-a4fd-575d6ab20bb8-kube-api-access-dqrmz\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-tp7s8\" (UID: \"059ef715-b1d1-47e5-a4fd-575d6ab20bb8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tp7s8"
Oct 31 03:23:58.973853 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:58.973819    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/059ef715-b1d1-47e5-a4fd-575d6ab20bb8-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-tp7s8\" (UID: \"059ef715-b1d1-47e5-a4fd-575d6ab20bb8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tp7s8"
Oct 31 03:23:59.074262 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:59.074227    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqrmz\" (UniqueName: \"kubernetes.io/projected/059ef715-b1d1-47e5-a4fd-575d6ab20bb8-kube-api-access-dqrmz\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-tp7s8\" (UID: \"059ef715-b1d1-47e5-a4fd-575d6ab20bb8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tp7s8"
Oct 31 03:23:59.074434 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:59.074310    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/059ef715-b1d1-47e5-a4fd-575d6ab20bb8-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-tp7s8\" (UID: \"059ef715-b1d1-47e5-a4fd-575d6ab20bb8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tp7s8"
Oct 31 03:23:59.074673 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:59.074653    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/059ef715-b1d1-47e5-a4fd-575d6ab20bb8-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-tp7s8\" (UID: \"059ef715-b1d1-47e5-a4fd-575d6ab20bb8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tp7s8"
Oct 31 03:23:59.136047 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:59.135978    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqrmz\" (UniqueName: \"kubernetes.io/projected/059ef715-b1d1-47e5-a4fd-575d6ab20bb8-kube-api-access-dqrmz\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-tp7s8\" (UID: \"059ef715-b1d1-47e5-a4fd-575d6ab20bb8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tp7s8"
Oct 31 03:23:59.170795 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:59.170762    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tp7s8"
Oct 31 03:23:59.592595 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:59.592571    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tp7s8"]
Oct 31 03:23:59.594623 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:23:59.594595    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059ef715_b1d1_47e5_a4fd_575d6ab20bb8.slice/crio-378cbeb46f8f54be4923777bb679f847a53cdf9912ce33c461955ecff304874c WatchSource:0}: Error finding container 378cbeb46f8f54be4923777bb679f847a53cdf9912ce33c461955ecff304874c: Status 404 returned error can't find the container with id 378cbeb46f8f54be4923777bb679f847a53cdf9912ce33c461955ecff304874c
Oct 31 03:23:59.974872 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:23:59.974840    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tp7s8" event={"ID":"059ef715-b1d1-47e5-a4fd-575d6ab20bb8","Type":"ContainerStarted","Data":"378cbeb46f8f54be4923777bb679f847a53cdf9912ce33c461955ecff304874c"}
Oct 31 03:24:02.990827 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:02.990792    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tp7s8" event={"ID":"059ef715-b1d1-47e5-a4fd-575d6ab20bb8","Type":"ContainerStarted","Data":"d60705d3ae4fe3db33cd5e2435150ab647e9620d9677c07c752d8b43cd7a4899"}
Oct 31 03:24:03.211769 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:03.211714    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-tp7s8" podStartSLOduration=2.163565806 podStartE2EDuration="5.211697505s" totalImagesPullingTime="3.048131699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:03.160631914 +0000 UTC m=+268.671614319" watchObservedRunningTime="2026-10-31 03:24:03.211697505 +0000 UTC m=+268.722679882"
Oct 31 03:24:05.953112 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:05.953078    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-275tg"]
Oct 31 03:24:05.955208 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:05.955190    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-275tg"
Oct 31 03:24:06.040856 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:06.040819    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k687b\" (UniqueName: \"kubernetes.io/projected/a5ebe756-cf88-441b-aba4-51bb5f9d1922-kube-api-access-k687b\") pod \"limitador-operator-controller-manager-5778bb4449-275tg\" (UID: \"a5ebe756-cf88-441b-aba4-51bb5f9d1922\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-275tg"
Oct 31 03:24:06.061326 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:06.061291    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 03:24:06.061493 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:06.061375    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 03:24:06.061621 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:06.061609    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-nnlv5\""
Oct 31 03:24:06.063675 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:06.063654    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-275tg"]
Oct 31 03:24:06.142299 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:06.142270    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k687b\" (UniqueName: \"kubernetes.io/projected/a5ebe756-cf88-441b-aba4-51bb5f9d1922-kube-api-access-k687b\") pod \"limitador-operator-controller-manager-5778bb4449-275tg\" (UID: \"a5ebe756-cf88-441b-aba4-51bb5f9d1922\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-275tg"
Oct 31 03:24:06.211477 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:06.211449    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k687b\" (UniqueName: \"kubernetes.io/projected/a5ebe756-cf88-441b-aba4-51bb5f9d1922-kube-api-access-k687b\") pod \"limitador-operator-controller-manager-5778bb4449-275tg\" (UID: \"a5ebe756-cf88-441b-aba4-51bb5f9d1922\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-275tg"
Oct 31 03:24:06.265976 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:06.265945    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-275tg"
Oct 31 03:24:06.643990 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:06.643962    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-275tg"]
Oct 31 03:24:06.645853 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:24:06.645826    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ebe756_cf88_441b_aba4_51bb5f9d1922.slice/crio-c60451be86f0d6b3eef79e7886253f26a6ec7b7421d3a20a6823f430da2fdc22 WatchSource:0}: Error finding container c60451be86f0d6b3eef79e7886253f26a6ec7b7421d3a20a6823f430da2fdc22: Status 404 returned error can't find the container with id c60451be86f0d6b3eef79e7886253f26a6ec7b7421d3a20a6823f430da2fdc22
Oct 31 03:24:07.010186 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:07.010154    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-275tg" event={"ID":"a5ebe756-cf88-441b-aba4-51bb5f9d1922","Type":"ContainerStarted","Data":"c60451be86f0d6b3eef79e7886253f26a6ec7b7421d3a20a6823f430da2fdc22"}
Oct 31 03:24:07.966684 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:07.966652    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84dfd97fcd-shc8p"
Oct 31 03:24:11.032821 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:11.032784    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-275tg" event={"ID":"a5ebe756-cf88-441b-aba4-51bb5f9d1922","Type":"ContainerStarted","Data":"25663d343961263a88a1e8f874059d18d74c0bf3fa21e3929163e094567398b3"}
Oct 31 03:24:11.033308 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:11.032913    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-275tg"
Oct 31 03:24:11.150735 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:11.150680    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-275tg" podStartSLOduration=2.3207347609999998 podStartE2EDuration="6.150665703s" totalImagesPullingTime="3.829930942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:11.149711503 +0000 UTC m=+276.660693880" watchObservedRunningTime="2026-10-31 03:24:11.150665703 +0000 UTC m=+276.661648080"
Oct 31 03:24:13.456039 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:13.456001    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hj86s"]
Oct 31 03:24:13.458267 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:13.458252    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hj86s"
Oct 31 03:24:13.505982 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:13.505954    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9fd9\" (UniqueName: \"kubernetes.io/projected/583b06d7-5204-42dd-91f3-5b54d779c5b8-kube-api-access-k9fd9\") pod \"kuadrant-operator-controller-manager-cf8469d7b-hj86s\" (UID: \"583b06d7-5204-42dd-91f3-5b54d779c5b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hj86s"
Oct 31 03:24:13.506127 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:13.506007    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/583b06d7-5204-42dd-91f3-5b54d779c5b8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-hj86s\" (UID: \"583b06d7-5204-42dd-91f3-5b54d779c5b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hj86s"
Oct 31 03:24:13.562760 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:13.562719    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-v67zc\""
Oct 31 03:24:13.564640 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:13.564612    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hj86s"]
Oct 31 03:24:13.607032 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:13.607003    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9fd9\" (UniqueName: \"kubernetes.io/projected/583b06d7-5204-42dd-91f3-5b54d779c5b8-kube-api-access-k9fd9\") pod \"kuadrant-operator-controller-manager-cf8469d7b-hj86s\" (UID: \"583b06d7-5204-42dd-91f3-5b54d779c5b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hj86s"
Oct 31 03:24:13.607171 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:13.607053    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/583b06d7-5204-42dd-91f3-5b54d779c5b8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-hj86s\" (UID: \"583b06d7-5204-42dd-91f3-5b54d779c5b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hj86s"
Oct 31 03:24:13.607421 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:13.607400    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/583b06d7-5204-42dd-91f3-5b54d779c5b8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-hj86s\" (UID: \"583b06d7-5204-42dd-91f3-5b54d779c5b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hj86s"
Oct 31 03:24:13.673046 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:13.673019    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9fd9\" (UniqueName: \"kubernetes.io/projected/583b06d7-5204-42dd-91f3-5b54d779c5b8-kube-api-access-k9fd9\") pod \"kuadrant-operator-controller-manager-cf8469d7b-hj86s\" (UID: \"583b06d7-5204-42dd-91f3-5b54d779c5b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hj86s"
Oct 31 03:24:13.769792 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:13.769705    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hj86s"
Oct 31 03:24:14.245103 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:14.245068    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hj86s"]
Oct 31 03:24:14.288821 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:24:14.288749    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583b06d7_5204_42dd_91f3_5b54d779c5b8.slice/crio-b0f07389cceff0d861cee7fa5a67c07774af2b0d1f8f40cc9a5dd7e395a11f66 WatchSource:0}: Error finding container b0f07389cceff0d861cee7fa5a67c07774af2b0d1f8f40cc9a5dd7e395a11f66: Status 404 returned error can't find the container with id b0f07389cceff0d861cee7fa5a67c07774af2b0d1f8f40cc9a5dd7e395a11f66
Oct 31 03:24:15.052753 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:15.052718    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hj86s" event={"ID":"583b06d7-5204-42dd-91f3-5b54d779c5b8","Type":"ContainerStarted","Data":"b0f07389cceff0d861cee7fa5a67c07774af2b0d1f8f40cc9a5dd7e395a11f66"}
Oct 31 03:24:17.884675 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:17.884637    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-mtv6q"]
Oct 31 03:24:17.887373 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:17.887350    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mtv6q"
Oct 31 03:24:17.943712 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:17.943685    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-6jsgw\""
Oct 31 03:24:17.943911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:17.943754    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 31 03:24:17.944235 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:17.944211    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j2g9\" (UniqueName: \"kubernetes.io/projected/f4fc7df3-3489-458c-971e-7f2717626680-kube-api-access-7j2g9\") pod \"dns-operator-controller-manager-8557cbdb84-mtv6q\" (UID: \"f4fc7df3-3489-458c-971e-7f2717626680\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mtv6q"
Oct 31 03:24:17.948672 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:17.948597    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-mtv6q"]
Oct 31 03:24:18.045400 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:18.045364    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j2g9\" (UniqueName: \"kubernetes.io/projected/f4fc7df3-3489-458c-971e-7f2717626680-kube-api-access-7j2g9\") pod \"dns-operator-controller-manager-8557cbdb84-mtv6q\" (UID: \"f4fc7df3-3489-458c-971e-7f2717626680\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mtv6q"
Oct 31 03:24:18.126991 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:18.126955    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j2g9\" (UniqueName: \"kubernetes.io/projected/f4fc7df3-3489-458c-971e-7f2717626680-kube-api-access-7j2g9\") pod \"dns-operator-controller-manager-8557cbdb84-mtv6q\" (UID: \"f4fc7df3-3489-458c-971e-7f2717626680\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mtv6q"
Oct 31 03:24:18.201045 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:18.200957    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mtv6q"
Oct 31 03:24:18.606138 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:24:18.606102    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4fc7df3_3489_458c_971e_7f2717626680.slice/crio-efb63abd7b06a7e77854bbe7744f1b67824cd110a981d3d1472513a129b3eb18 WatchSource:0}: Error finding container efb63abd7b06a7e77854bbe7744f1b67824cd110a981d3d1472513a129b3eb18: Status 404 returned error can't find the container with id efb63abd7b06a7e77854bbe7744f1b67824cd110a981d3d1472513a129b3eb18
Oct 31 03:24:18.609128 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:18.608841    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-mtv6q"]
Oct 31 03:24:19.010760 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:24:19.010718    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:24:19.080038 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:19.079918    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mtv6q" event={"ID":"f4fc7df3-3489-458c-971e-7f2717626680","Type":"ContainerStarted","Data":"efb63abd7b06a7e77854bbe7744f1b67824cd110a981d3d1472513a129b3eb18"}
Oct 31 03:24:20.086601 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:20.086560    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hj86s" event={"ID":"583b06d7-5204-42dd-91f3-5b54d779c5b8","Type":"ContainerStarted","Data":"f5b9eb7f48e7101d38c880d2acf9dbc75a3577b83b4607eb35c5fc5078cbf0b3"}
Oct 31 03:24:20.087007 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:20.086783    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hj86s"
Oct 31 03:24:20.265208 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:20.265145    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hj86s" podStartSLOduration=1.6212578500000001 podStartE2EDuration="7.265128151s" totalImagesPullingTime="5.643870301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:20.265111805 +0000 UTC m=+285.776094183" watchObservedRunningTime="2026-10-31 03:24:20.265128151 +0000 UTC m=+285.776110529"
Oct 31 03:24:22.039294 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:22.039265    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-275tg"
Oct 31 03:24:22.253092 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:22.253062    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-4bnf2"]
Oct 31 03:24:22.256520 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:22.256498    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4bnf2"
Oct 31 03:24:22.320593 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:22.320506    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-4bnf2"]
Oct 31 03:24:22.349220 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:22.349193    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-6rmmh\""
Oct 31 03:24:22.385716 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:22.385678    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7tf9\" (UniqueName: \"kubernetes.io/projected/9670f62f-e977-4370-951a-2a7bcb9209ac-kube-api-access-s7tf9\") pod \"authorino-operator-7b7bb4bc77-4bnf2\" (UID: \"9670f62f-e977-4370-951a-2a7bcb9209ac\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-4bnf2"
Oct 31 03:24:22.486530 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:22.486501    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7tf9\" (UniqueName: \"kubernetes.io/projected/9670f62f-e977-4370-951a-2a7bcb9209ac-kube-api-access-s7tf9\") pod \"authorino-operator-7b7bb4bc77-4bnf2\" (UID: \"9670f62f-e977-4370-951a-2a7bcb9209ac\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-4bnf2"
Oct 31 03:24:22.547409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:22.547383    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7tf9\" (UniqueName: \"kubernetes.io/projected/9670f62f-e977-4370-951a-2a7bcb9209ac-kube-api-access-s7tf9\") pod \"authorino-operator-7b7bb4bc77-4bnf2\" (UID: \"9670f62f-e977-4370-951a-2a7bcb9209ac\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-4bnf2"
Oct 31 03:24:22.571155 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:22.571112    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4bnf2"
Oct 31 03:24:22.950959 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:22.948995    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-4bnf2"]
Oct 31 03:24:22.953467 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:24:22.953432    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9670f62f_e977_4370_951a_2a7bcb9209ac.slice/crio-183928c0f6e4af85850d80df9a993edf31214f4f1dbc34c08f30479944f5a099 WatchSource:0}: Error finding container 183928c0f6e4af85850d80df9a993edf31214f4f1dbc34c08f30479944f5a099: Status 404 returned error can't find the container with id 183928c0f6e4af85850d80df9a993edf31214f4f1dbc34c08f30479944f5a099
Oct 31 03:24:23.102673 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:23.102635    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mtv6q" event={"ID":"f4fc7df3-3489-458c-971e-7f2717626680","Type":"ContainerStarted","Data":"ad544ee0fcf15bc614ebe1e48d10bc46f0b24693853f8332aa880eb8a9414052"}
Oct 31 03:24:23.103187 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:23.102757    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mtv6q"
Oct 31 03:24:23.104028 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:23.104001    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4bnf2" event={"ID":"9670f62f-e977-4370-951a-2a7bcb9209ac","Type":"ContainerStarted","Data":"183928c0f6e4af85850d80df9a993edf31214f4f1dbc34c08f30479944f5a099"}
Oct 31 03:24:23.249935 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:23.249858    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mtv6q" podStartSLOduration=2.341003111 podStartE2EDuration="6.249836421s" totalImagesPullingTime="3.90883331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:23.246518863 +0000 UTC m=+288.757501246" watchObservedRunningTime="2026-10-31 03:24:23.249836421 +0000 UTC m=+288.760818799"
Oct 31 03:24:23.921235 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:23.921200    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"]
Oct 31 03:24:23.925794 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:23.925772    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:23.982016 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:23.981986    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-qq6xx\""
Oct 31 03:24:24.002252 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.002225    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.002252 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.002254    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.002446 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.002281    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.002446 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.002339    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.002446 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.002374    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.002568 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.002470    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.002568 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.002538    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.002669 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.002621    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.002669 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.002659    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n96jv\" (UniqueName: \"kubernetes.io/projected/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-kube-api-access-n96jv\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.007504 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.007476    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"]
Oct 31 03:24:24.103843 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.103808    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.103843 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.103842    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n96jv\" (UniqueName: \"kubernetes.io/projected/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-kube-api-access-n96jv\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.104409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.103877    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.104409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.103948    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.104409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.103992    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.104409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.104015    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.104409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.104037    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.104409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.104081    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.104409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.104180    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.104409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.104378    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.104810 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.104494    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.104810 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.104557    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.104810 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.104638    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.104810 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.104732    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.106522 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.106496    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.106837 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.106817    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.169597 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.169563    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n96jv\" (UniqueName: \"kubernetes.io/projected/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-kube-api-access-n96jv\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.169597 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.169574    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5de1f7a-3665-4f3c-9f23-3febe52c88d0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7\" (UID: \"c5de1f7a-3665-4f3c-9f23-3febe52c88d0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.239758 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.239725    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:24.668295 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.668270    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"]
Oct 31 03:24:24.669963 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:24:24.669930    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5de1f7a_3665_4f3c_9f23_3febe52c88d0.slice/crio-993c6735332c7243a9cd23a0f06894c6ac15b21710c8da84025a11a9f4841b51 WatchSource:0}: Error finding container 993c6735332c7243a9cd23a0f06894c6ac15b21710c8da84025a11a9f4841b51: Status 404 returned error can't find the container with id 993c6735332c7243a9cd23a0f06894c6ac15b21710c8da84025a11a9f4841b51
Oct 31 03:24:24.672099 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.672065    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:24:24.672200 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.672145    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:24:24.672200 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:24.672190    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:24:25.116415 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:25.116378    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7" event={"ID":"c5de1f7a-3665-4f3c-9f23-3febe52c88d0","Type":"ContainerStarted","Data":"335e5ca400ede68443c453ac28bb9a705b2a7f194acf096a7fefdcc9e15aa128"}
Oct 31 03:24:25.116822 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:25.116443    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7" event={"ID":"c5de1f7a-3665-4f3c-9f23-3febe52c88d0","Type":"ContainerStarted","Data":"993c6735332c7243a9cd23a0f06894c6ac15b21710c8da84025a11a9f4841b51"}
Oct 31 03:24:25.240107 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:25.240078    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:25.245464 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:25.245437    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:25.370727 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:25.370629    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7" podStartSLOduration=2.370611367 podStartE2EDuration="2.370611367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:25.240100152 +0000 UTC m=+290.751082529" watchObservedRunningTime="2026-10-31 03:24:25.370611367 +0000 UTC m=+290.881593744"
Oct 31 03:24:26.123458 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:26.123409    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4bnf2" event={"ID":"9670f62f-e977-4370-951a-2a7bcb9209ac","Type":"ContainerStarted","Data":"ba3d9dd58743b4ff268e9f83bd35db20d187d28abda109dc22a520b3814d5bdd"}
Oct 31 03:24:26.123963 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:26.123718    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4bnf2"
Oct 31 03:24:26.123963 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:26.123754    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:26.124798 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:26.124779    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7"
Oct 31 03:24:26.259746 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:26.259672    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4bnf2" podStartSLOduration=1.714678172 podStartE2EDuration="4.25964919s" totalImagesPullingTime="2.544971018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:26.25896624 +0000 UTC m=+291.769948620" watchObservedRunningTime="2026-10-31 03:24:26.25964919 +0000 UTC m=+291.770631567"
Oct 31 03:24:26.881840 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:26.881806    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-bznww"]
Oct 31 03:24:26.885586 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:26.885566    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-bznww"
Oct 31 03:24:26.942161 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:26.942127    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 31 03:24:26.942760 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:26.942737    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2mggw\""
Oct 31 03:24:26.944688 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:26.944667    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-bznww"]
Oct 31 03:24:27.030664 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:27.030629    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twz64\" (UniqueName: \"kubernetes.io/projected/b3127dc5-e686-4f9a-864a-6092f5663790-kube-api-access-twz64\") pod \"limitador-limitador-74dfcfb894-bznww\" (UID: \"b3127dc5-e686-4f9a-864a-6092f5663790\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-bznww"
Oct 31 03:24:27.030817 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:27.030693    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b3127dc5-e686-4f9a-864a-6092f5663790-config-file\") pod \"limitador-limitador-74dfcfb894-bznww\" (UID: \"b3127dc5-e686-4f9a-864a-6092f5663790\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-bznww"
Oct 31 03:24:27.130590 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:27.130548    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-bznww"]
Oct 31 03:24:27.131440 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:27.131351    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twz64\" (UniqueName: \"kubernetes.io/projected/b3127dc5-e686-4f9a-864a-6092f5663790-kube-api-access-twz64\") pod \"limitador-limitador-74dfcfb894-bznww\" (UID: \"b3127dc5-e686-4f9a-864a-6092f5663790\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-bznww"
Oct 31 03:24:27.131589 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:27.131565    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b3127dc5-e686-4f9a-864a-6092f5663790-config-file\") pod \"limitador-limitador-74dfcfb894-bznww\" (UID: \"b3127dc5-e686-4f9a-864a-6092f5663790\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-bznww"
Oct 31 03:24:27.132456 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:27.132400    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b3127dc5-e686-4f9a-864a-6092f5663790-config-file\") pod \"limitador-limitador-74dfcfb894-bznww\" (UID: \"b3127dc5-e686-4f9a-864a-6092f5663790\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-bznww"
Oct 31 03:24:27.196100 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:27.196073    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twz64\" (UniqueName: \"kubernetes.io/projected/b3127dc5-e686-4f9a-864a-6092f5663790-kube-api-access-twz64\") pod \"limitador-limitador-74dfcfb894-bznww\" (UID: \"b3127dc5-e686-4f9a-864a-6092f5663790\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-bznww"
Oct 31 03:24:27.198952 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:27.198925    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-bznww"
Oct 31 03:24:27.580430 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:27.580406    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-bznww"]
Oct 31 03:24:27.582530 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:24:27.582499    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3127dc5_e686_4f9a_864a_6092f5663790.slice/crio-eb560d80fb6f2d3c9831bf350f67429c3edd6541bc00f118434dd3d4b1e1cc15 WatchSource:0}: Error finding container eb560d80fb6f2d3c9831bf350f67429c3edd6541bc00f118434dd3d4b1e1cc15: Status 404 returned error can't find the container with id eb560d80fb6f2d3c9831bf350f67429c3edd6541bc00f118434dd3d4b1e1cc15
Oct 31 03:24:28.136563 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:28.136525    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-bznww" event={"ID":"b3127dc5-e686-4f9a-864a-6092f5663790","Type":"ContainerStarted","Data":"eb560d80fb6f2d3c9831bf350f67429c3edd6541bc00f118434dd3d4b1e1cc15"}
Oct 31 03:24:29.825027 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:29.824992    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-skhfw"]
Oct 31 03:24:29.827616 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:29.827588    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-skhfw"
Oct 31 03:24:29.886065 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:29.886031    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-skhfw"]
Oct 31 03:24:29.932726 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:29.932698    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bq6cs\""
Oct 31 03:24:29.960021 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:29.959995    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcjd7\" (UniqueName: \"kubernetes.io/projected/366b75db-ad1b-4ab3-b8c2-0b99cd8c0568-kube-api-access-qcjd7\") pod \"authorino-56c6c9ddd5-skhfw\" (UID: \"366b75db-ad1b-4ab3-b8c2-0b99cd8c0568\") " pod="kuadrant-system/authorino-56c6c9ddd5-skhfw"
Oct 31 03:24:30.005657 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:30.005622    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhpht"]
Oct 31 03:24:30.007843 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:30.007828    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhpht"
Oct 31 03:24:30.061366 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:30.061337    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcjd7\" (UniqueName: \"kubernetes.io/projected/366b75db-ad1b-4ab3-b8c2-0b99cd8c0568-kube-api-access-qcjd7\") pod \"authorino-56c6c9ddd5-skhfw\" (UID: \"366b75db-ad1b-4ab3-b8c2-0b99cd8c0568\") " pod="kuadrant-system/authorino-56c6c9ddd5-skhfw"
Oct 31 03:24:30.066369 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:30.066343    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 31 03:24:30.066369 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:30.066359    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 31 03:24:30.068494 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:30.068471    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhpht"]
Oct 31 03:24:30.123042 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:30.122972    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcjd7\" (UniqueName: \"kubernetes.io/projected/366b75db-ad1b-4ab3-b8c2-0b99cd8c0568-kube-api-access-qcjd7\") pod \"authorino-56c6c9ddd5-skhfw\" (UID: \"366b75db-ad1b-4ab3-b8c2-0b99cd8c0568\") " pod="kuadrant-system/authorino-56c6c9ddd5-skhfw"
Oct 31 03:24:30.154312 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:30.154281    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-skhfw"
Oct 31 03:24:30.154553 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:30.154533    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-bznww" event={"ID":"b3127dc5-e686-4f9a-864a-6092f5663790","Type":"ContainerStarted","Data":"1ce940ca2b0678861492cc271d90376f3e21fee8df2cae847a66e371a983e8ca"}
Oct 31 03:24:30.154698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:30.154684    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-bznww"
Oct 31 03:24:30.162493 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:30.162461    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dd0803af-c8ee-4ae0-8082-1c4e47742909-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vhpht\" (UID: \"dd0803af-c8ee-4ae0-8082-1c4e47742909\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhpht"
Oct 31 03:24:30.162628 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:30.162512    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd0803af-c8ee-4ae0-8082-1c4e47742909-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vhpht\" (UID: \"dd0803af-c8ee-4ae0-8082-1c4e47742909\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhpht"
Oct 31 03:24:30.162628 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:30.162581    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl55r\" (UniqueName: \"kubernetes.io/projected/dd0803af-c8ee-4ae0-8082-1c4e47742909-kube-api-access-jl55r\") pod \"kuadrant-console-plugin-66c4fb575d-vhpht\" (UID: \"dd0803af-c8ee-4ae0-8082-1c4e47742909\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhpht"
Oct 31 03:24:30.263938 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:30.263870    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dd0803af-c8ee-4ae0-8082-1c4e47742909-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vhpht\" (UID: \"dd0803af-c8ee-4ae0-8082-1c4e47742909\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhpht"
Oct 31 03:24:30.264110 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:30.263953    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd0803af-c8ee-4ae0-8082-1c4e47742909-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vhpht\" (UID: \"dd0803af-c8ee-4ae0-8082-1c4e47742909\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhpht"
Oct 31 03:24:30.264110 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:30.264009    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl55r\" (UniqueName: \"kubernetes.io/projected/dd0803af-c8ee-4ae0-8082-1c4e47742909-kube-api-access-jl55r\") pod \"kuadrant-console-plugin-66c4fb575d-vhpht\" (UID: \"dd0803af-c8ee-4ae0-8082-1c4e47742909\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhpht"
Oct 31 03:24:30.264595 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:30.264572    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dd0803af-c8ee-4ae0-8082-1c4e47742909-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vhpht\" (UID: \"dd0803af-c8ee-4ae0-8082-1c4e47742909\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhpht"
Oct 31 03:24:30.266586 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:30.266563    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/dd0803af-c8ee-4ae0-8082-1c4e47742909-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vhpht\" (UID: \"dd0803af-c8ee-4ae0-8082-1c4e47742909\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhpht"
Oct 31 03:24:30.273319 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:30.273269    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-bznww" podStartSLOduration=1.934895145 podStartE2EDuration="4.273253305s" totalImagesPullingTime="2.33835816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:30.271941531 +0000 UTC m=+295.782923908" watchObservedRunningTime="2026-10-31 03:24:30.273253305 +0000 UTC m=+295.784235680"
Oct 31 03:24:30.328436 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:30.328398    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl55r\" (UniqueName: \"kubernetes.io/projected/dd0803af-c8ee-4ae0-8082-1c4e47742909-kube-api-access-jl55r\") pod \"kuadrant-console-plugin-66c4fb575d-vhpht\" (UID: \"dd0803af-c8ee-4ae0-8082-1c4e47742909\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhpht"
Oct 31 03:24:30.522591 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:30.522565    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-skhfw"]
Oct 31 03:24:30.524149 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:24:30.524116    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod366b75db_ad1b_4ab3_b8c2_0b99cd8c0568.slice/crio-9fd9e0a8af4ad9449fd571b896fdcdbe46a4a2e642944a8f4698ee692420931b WatchSource:0}: Error finding container 9fd9e0a8af4ad9449fd571b896fdcdbe46a4a2e642944a8f4698ee692420931b: Status 404 returned error can't find the container with id 9fd9e0a8af4ad9449fd571b896fdcdbe46a4a2e642944a8f4698ee692420931b
Oct 31 03:24:30.618299 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:30.618268    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhpht"
Oct 31 03:24:30.988183 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:30.988160    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhpht"]
Oct 31 03:24:30.989980 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:24:30.989953    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd0803af_c8ee_4ae0_8082_1c4e47742909.slice/crio-7c7aacb98e87188b15537dcecebe1440c7228c89b9a0250a88fc7b9f1d40c236 WatchSource:0}: Error finding container 7c7aacb98e87188b15537dcecebe1440c7228c89b9a0250a88fc7b9f1d40c236: Status 404 returned error can't find the container with id 7c7aacb98e87188b15537dcecebe1440c7228c89b9a0250a88fc7b9f1d40c236
Oct 31 03:24:31.094442 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:31.094417    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hj86s"
Oct 31 03:24:31.161312 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:31.161277    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhpht" event={"ID":"dd0803af-c8ee-4ae0-8082-1c4e47742909","Type":"ContainerStarted","Data":"7c7aacb98e87188b15537dcecebe1440c7228c89b9a0250a88fc7b9f1d40c236"}
Oct 31 03:24:31.162490 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:31.162467    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-skhfw" event={"ID":"366b75db-ad1b-4ab3-b8c2-0b99cd8c0568","Type":"ContainerStarted","Data":"9fd9e0a8af4ad9449fd571b896fdcdbe46a4a2e642944a8f4698ee692420931b"}
Oct 31 03:24:33.180614 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:33.180576    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-skhfw" event={"ID":"366b75db-ad1b-4ab3-b8c2-0b99cd8c0568","Type":"ContainerStarted","Data":"509b52ab1bb39903ce6edf7f0719f74c0f361ca8d050e01d77c270d239df489a"}
Oct 31 03:24:33.296967 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:33.296566    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-skhfw" podStartSLOduration=1.835761287 podStartE2EDuration="4.296546656s" totalImagesPullingTime="2.460785369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:33.296225497 +0000 UTC m=+298.807207876" watchObservedRunningTime="2026-10-31 03:24:33.296546656 +0000 UTC m=+298.807529036"
Oct 31 03:24:34.111303 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:34.111265    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-mtv6q"
Oct 31 03:24:34.644455 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:34.644423    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 03:24:34.644936 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:34.644490    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 03:24:37.133046 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:37.133017    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4bnf2"
Oct 31 03:24:37.203397 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:37.203367    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhpht" event={"ID":"dd0803af-c8ee-4ae0-8082-1c4e47742909","Type":"ContainerStarted","Data":"fe99f69515d9d283a4c1d3094eaca7e1d4b4c2796dd1832eddbedea0fdb7c3d7"}
Oct 31 03:24:37.387551 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:37.387441    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vhpht" podStartSLOduration=2.415973054 podStartE2EDuration="8.387426072s" totalImagesPullingTime="5.971453018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:37.386707175 +0000 UTC m=+302.897689564" watchObservedRunningTime="2026-10-31 03:24:37.387426072 +0000 UTC m=+302.898408449"
Oct 31 03:24:41.163380 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:41.163349    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-bznww"
Oct 31 03:24:44.647941 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:44.647905    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-bznww"]
Oct 31 03:24:44.648419 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:44.648085    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-bznww" podUID="b3127dc5-e686-4f9a-864a-6092f5663790" containerName="limitador" containerID="cri-o://1ce940ca2b0678861492cc271d90376f3e21fee8df2cae847a66e371a983e8ca" gracePeriod=30
Oct 31 03:24:45.206091 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:45.206067    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-bznww"
Oct 31 03:24:45.243896 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:45.243796    4447 generic.go:356] "Generic (PLEG): container finished" podID="b3127dc5-e686-4f9a-864a-6092f5663790" containerID="1ce940ca2b0678861492cc271d90376f3e21fee8df2cae847a66e371a983e8ca" exitCode=0
Oct 31 03:24:45.243896 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:45.243841    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-bznww"
Oct 31 03:24:45.244082 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:45.243908    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-bznww" event={"ID":"b3127dc5-e686-4f9a-864a-6092f5663790","Type":"ContainerDied","Data":"1ce940ca2b0678861492cc271d90376f3e21fee8df2cae847a66e371a983e8ca"}
Oct 31 03:24:45.244082 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:45.243964    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-bznww" event={"ID":"b3127dc5-e686-4f9a-864a-6092f5663790","Type":"ContainerDied","Data":"eb560d80fb6f2d3c9831bf350f67429c3edd6541bc00f118434dd3d4b1e1cc15"}
Oct 31 03:24:45.244082 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:45.243992    4447 scope.go:122] "RemoveContainer" containerID="1ce940ca2b0678861492cc271d90376f3e21fee8df2cae847a66e371a983e8ca"
Oct 31 03:24:45.252725 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:45.252699    4447 scope.go:122] "RemoveContainer" containerID="1ce940ca2b0678861492cc271d90376f3e21fee8df2cae847a66e371a983e8ca"
Oct 31 03:24:45.253034 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:24:45.253012    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ce940ca2b0678861492cc271d90376f3e21fee8df2cae847a66e371a983e8ca\": container with ID starting with 1ce940ca2b0678861492cc271d90376f3e21fee8df2cae847a66e371a983e8ca not found: ID does not exist" containerID="1ce940ca2b0678861492cc271d90376f3e21fee8df2cae847a66e371a983e8ca"
Oct 31 03:24:45.253117 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:45.253045    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce940ca2b0678861492cc271d90376f3e21fee8df2cae847a66e371a983e8ca"} err="failed to get container status \"1ce940ca2b0678861492cc271d90376f3e21fee8df2cae847a66e371a983e8ca\": rpc error: code = NotFound desc = could not find container \"1ce940ca2b0678861492cc271d90376f3e21fee8df2cae847a66e371a983e8ca\": container with ID starting with 1ce940ca2b0678861492cc271d90376f3e21fee8df2cae847a66e371a983e8ca not found: ID does not exist"
Oct 31 03:24:45.314126 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:45.314093    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3127dc5-e686-4f9a-864a-6092f5663790-config-file\" (UniqueName: \"kubernetes.io/configmap/b3127dc5-e686-4f9a-864a-6092f5663790-config-file\") pod \"b3127dc5-e686-4f9a-864a-6092f5663790\" (UID: \"b3127dc5-e686-4f9a-864a-6092f5663790\") "
Oct 31 03:24:45.314307 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:45.314189    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3127dc5-e686-4f9a-864a-6092f5663790-kube-api-access-twz64\" (UniqueName: \"kubernetes.io/projected/b3127dc5-e686-4f9a-864a-6092f5663790-kube-api-access-twz64\") pod \"b3127dc5-e686-4f9a-864a-6092f5663790\" (UID: \"b3127dc5-e686-4f9a-864a-6092f5663790\") "
Oct 31 03:24:45.314504 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:45.314478    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3127dc5-e686-4f9a-864a-6092f5663790-config-file" pod "b3127dc5-e686-4f9a-864a-6092f5663790" (UID: "b3127dc5-e686-4f9a-864a-6092f5663790"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:24:45.316535 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:45.316508    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3127dc5-e686-4f9a-864a-6092f5663790-kube-api-access-twz64" pod "b3127dc5-e686-4f9a-864a-6092f5663790" (UID: "b3127dc5-e686-4f9a-864a-6092f5663790"). InnerVolumeSpecName "kube-api-access-twz64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:24:45.415540 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:45.415509    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twz64\" (UniqueName: \"kubernetes.io/projected/b3127dc5-e686-4f9a-864a-6092f5663790-kube-api-access-twz64\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:24:45.415540 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:45.415537    4447 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b3127dc5-e686-4f9a-864a-6092f5663790-config-file\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:24:45.663547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:45.663514    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-bznww"]
Oct 31 03:24:45.668640 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:45.668614    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-bznww"]
Oct 31 03:24:46.606790 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:46.606715    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3127dc5-e686-4f9a-864a-6092f5663790" path="/var/lib/kubelet/pods/b3127dc5-e686-4f9a-864a-6092f5663790/volumes"
Oct 31 03:24:49.000161 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:24:49.000125    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:24:49.429600 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:49.429520    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hj86s"]
Oct 31 03:24:49.429783 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:49.429757    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hj86s" podUID="583b06d7-5204-42dd-91f3-5b54d779c5b8" containerName="manager" containerID="cri-o://f5b9eb7f48e7101d38c880d2acf9dbc75a3577b83b4607eb35c5fc5078cbf0b3" gracePeriod=10
Oct 31 03:24:49.673223 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:49.673200    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hj86s"
Oct 31 03:24:49.757622 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:49.757592    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583b06d7-5204-42dd-91f3-5b54d779c5b8-kube-api-access-k9fd9\" (UniqueName: \"kubernetes.io/projected/583b06d7-5204-42dd-91f3-5b54d779c5b8-kube-api-access-k9fd9\") pod \"583b06d7-5204-42dd-91f3-5b54d779c5b8\" (UID: \"583b06d7-5204-42dd-91f3-5b54d779c5b8\") "
Oct 31 03:24:49.757786 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:49.757632    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583b06d7-5204-42dd-91f3-5b54d779c5b8-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/583b06d7-5204-42dd-91f3-5b54d779c5b8-extensions-socket-volume\") pod \"583b06d7-5204-42dd-91f3-5b54d779c5b8\" (UID: \"583b06d7-5204-42dd-91f3-5b54d779c5b8\") "
Oct 31 03:24:49.758100 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:49.758070    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583b06d7-5204-42dd-91f3-5b54d779c5b8-extensions-socket-volume" pod "583b06d7-5204-42dd-91f3-5b54d779c5b8" (UID: "583b06d7-5204-42dd-91f3-5b54d779c5b8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:24:49.759835 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:49.759810    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583b06d7-5204-42dd-91f3-5b54d779c5b8-kube-api-access-k9fd9" pod "583b06d7-5204-42dd-91f3-5b54d779c5b8" (UID: "583b06d7-5204-42dd-91f3-5b54d779c5b8"). InnerVolumeSpecName "kube-api-access-k9fd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:24:49.859104 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:49.859069    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9fd9\" (UniqueName: \"kubernetes.io/projected/583b06d7-5204-42dd-91f3-5b54d779c5b8-kube-api-access-k9fd9\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:24:49.859104 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:49.859101    4447 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/583b06d7-5204-42dd-91f3-5b54d779c5b8-extensions-socket-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:24:49.863279 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:49.863246    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-nnpjm"]
Oct 31 03:24:49.863642 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:49.863630    4447 cpu_manager.go:414] "containerMap: removing container" podUID="583b06d7-5204-42dd-91f3-5b54d779c5b8" containerName="manager"
Oct 31 03:24:49.863685 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:49.863643    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583b06d7-5204-42dd-91f3-5b54d779c5b8" containerName="manager"
Oct 31 03:24:49.863685 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:49.863658    4447 cpu_manager.go:414] "containerMap: removing container" podUID="b3127dc5-e686-4f9a-864a-6092f5663790" containerName="limitador"
Oct 31 03:24:49.863685 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:49.863663    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3127dc5-e686-4f9a-864a-6092f5663790" containerName="limitador"
Oct 31 03:24:49.863768 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:49.863728    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="583b06d7-5204-42dd-91f3-5b54d779c5b8" containerName="manager"
Oct 31 03:24:49.863768 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:49.863736    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3127dc5-e686-4f9a-864a-6092f5663790" containerName="limitador"
Oct 31 03:24:49.867014 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:49.866998    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nnpjm"
Oct 31 03:24:49.925224 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:49.925200    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-8jmx9\""
Oct 31 03:24:49.932001 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:49.931975    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 31 03:24:49.933500 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:49.933475    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-nnpjm"]
Oct 31 03:24:49.940923 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:49.940865    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:24:49.941163 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:49.941148    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:24:49.960288 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:49.960255    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjb6z\" (UniqueName: \"kubernetes.io/projected/290bb433-86e4-4bcb-b609-abb84f34705f-kube-api-access-xjb6z\") pod \"postgres-7cdbf6dc44-nnpjm\" (UID: \"290bb433-86e4-4bcb-b609-abb84f34705f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nnpjm"
Oct 31 03:24:49.960381 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:49.960295    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/290bb433-86e4-4bcb-b609-abb84f34705f-data\") pod \"postgres-7cdbf6dc44-nnpjm\" (UID: \"290bb433-86e4-4bcb-b609-abb84f34705f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nnpjm"
Oct 31 03:24:50.061544 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.061467    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjb6z\" (UniqueName: \"kubernetes.io/projected/290bb433-86e4-4bcb-b609-abb84f34705f-kube-api-access-xjb6z\") pod \"postgres-7cdbf6dc44-nnpjm\" (UID: \"290bb433-86e4-4bcb-b609-abb84f34705f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nnpjm"
Oct 31 03:24:50.061544 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.061498    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/290bb433-86e4-4bcb-b609-abb84f34705f-data\") pod \"postgres-7cdbf6dc44-nnpjm\" (UID: \"290bb433-86e4-4bcb-b609-abb84f34705f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nnpjm"
Oct 31 03:24:50.062004 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.061857    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/290bb433-86e4-4bcb-b609-abb84f34705f-data\") pod \"postgres-7cdbf6dc44-nnpjm\" (UID: \"290bb433-86e4-4bcb-b609-abb84f34705f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nnpjm"
Oct 31 03:24:50.125015 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.124977    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjb6z\" (UniqueName: \"kubernetes.io/projected/290bb433-86e4-4bcb-b609-abb84f34705f-kube-api-access-xjb6z\") pod \"postgres-7cdbf6dc44-nnpjm\" (UID: \"290bb433-86e4-4bcb-b609-abb84f34705f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nnpjm"
Oct 31 03:24:50.179969 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.179937    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nnpjm"
Oct 31 03:24:50.279398 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.279035    4447 generic.go:356] "Generic (PLEG): container finished" podID="583b06d7-5204-42dd-91f3-5b54d779c5b8" containerID="f5b9eb7f48e7101d38c880d2acf9dbc75a3577b83b4607eb35c5fc5078cbf0b3" exitCode=0
Oct 31 03:24:50.279398 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.279146    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hj86s" event={"ID":"583b06d7-5204-42dd-91f3-5b54d779c5b8","Type":"ContainerDied","Data":"f5b9eb7f48e7101d38c880d2acf9dbc75a3577b83b4607eb35c5fc5078cbf0b3"}
Oct 31 03:24:50.279398 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.279186    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hj86s" event={"ID":"583b06d7-5204-42dd-91f3-5b54d779c5b8","Type":"ContainerDied","Data":"b0f07389cceff0d861cee7fa5a67c07774af2b0d1f8f40cc9a5dd7e395a11f66"}
Oct 31 03:24:50.279398 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.279207    4447 scope.go:122] "RemoveContainer" containerID="f5b9eb7f48e7101d38c880d2acf9dbc75a3577b83b4607eb35c5fc5078cbf0b3"
Oct 31 03:24:50.279398 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.279310    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hj86s"
Oct 31 03:24:50.295916 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.295777    4447 scope.go:122] "RemoveContainer" containerID="f5b9eb7f48e7101d38c880d2acf9dbc75a3577b83b4607eb35c5fc5078cbf0b3"
Oct 31 03:24:50.296203 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:24:50.296180    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5b9eb7f48e7101d38c880d2acf9dbc75a3577b83b4607eb35c5fc5078cbf0b3\": container with ID starting with f5b9eb7f48e7101d38c880d2acf9dbc75a3577b83b4607eb35c5fc5078cbf0b3 not found: ID does not exist" containerID="f5b9eb7f48e7101d38c880d2acf9dbc75a3577b83b4607eb35c5fc5078cbf0b3"
Oct 31 03:24:50.296268 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.296212    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b9eb7f48e7101d38c880d2acf9dbc75a3577b83b4607eb35c5fc5078cbf0b3"} err="failed to get container status \"f5b9eb7f48e7101d38c880d2acf9dbc75a3577b83b4607eb35c5fc5078cbf0b3\": rpc error: code = NotFound desc = could not find container \"f5b9eb7f48e7101d38c880d2acf9dbc75a3577b83b4607eb35c5fc5078cbf0b3\": container with ID starting with f5b9eb7f48e7101d38c880d2acf9dbc75a3577b83b4607eb35c5fc5078cbf0b3 not found: ID does not exist"
Oct 31 03:24:50.432850 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.432754    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-8ws7p"]
Oct 31 03:24:50.464772 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.464741    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hj86s"]
Oct 31 03:24:50.464772 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.464779    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-hj86s"]
Oct 31 03:24:50.465001 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.464821    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8ws7p"
Oct 31 03:24:50.501093 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.501064    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-8ws7p"]
Oct 31 03:24:50.534324 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.534213    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-qxrrg\""
Oct 31 03:24:50.534324 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.534278    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 03:24:50.534324 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.534280    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 03:24:50.570480 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.570448    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/90781578-9f71-4512-8141-d08c8830483b-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-8ws7p\" (UID: \"90781578-9f71-4512-8141-d08c8830483b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8ws7p"
Oct 31 03:24:50.570637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.570508    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n22kj\" (UniqueName: \"kubernetes.io/projected/90781578-9f71-4512-8141-d08c8830483b-kube-api-access-n22kj\") pod \"kserve-module-controller-manager-7566cf5fc8-8ws7p\" (UID: \"90781578-9f71-4512-8141-d08c8830483b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8ws7p"
Oct 31 03:24:50.570637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.570566    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90781578-9f71-4512-8141-d08c8830483b-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-8ws7p\" (UID: \"90781578-9f71-4512-8141-d08c8830483b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8ws7p"
Oct 31 03:24:50.570710 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.570696    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/90781578-9f71-4512-8141-d08c8830483b-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-8ws7p\" (UID: \"90781578-9f71-4512-8141-d08c8830483b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8ws7p"
Oct 31 03:24:50.584496 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.584468    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 31 03:24:50.605428 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:24:50.605401    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290bb433_86e4_4bcb_b609_abb84f34705f.slice/crio-615a31218d9498ba911f0bdc8ff323ae89eb1dfcdb8dfd3eea9d4ee21c661f3e WatchSource:0}: Error finding container 615a31218d9498ba911f0bdc8ff323ae89eb1dfcdb8dfd3eea9d4ee21c661f3e: Status 404 returned error can't find the container with id 615a31218d9498ba911f0bdc8ff323ae89eb1dfcdb8dfd3eea9d4ee21c661f3e
Oct 31 03:24:50.606914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.606859    4447 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:24:50.615048 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.610151    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="583b06d7-5204-42dd-91f3-5b54d779c5b8" path="/var/lib/kubelet/pods/583b06d7-5204-42dd-91f3-5b54d779c5b8/volumes"
Oct 31 03:24:50.615048 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.610498    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-nnpjm"]
Oct 31 03:24:50.671970 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.671938    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/90781578-9f71-4512-8141-d08c8830483b-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-8ws7p\" (UID: \"90781578-9f71-4512-8141-d08c8830483b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8ws7p"
Oct 31 03:24:50.672127 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.672011    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n22kj\" (UniqueName: \"kubernetes.io/projected/90781578-9f71-4512-8141-d08c8830483b-kube-api-access-n22kj\") pod \"kserve-module-controller-manager-7566cf5fc8-8ws7p\" (UID: \"90781578-9f71-4512-8141-d08c8830483b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8ws7p"
Oct 31 03:24:50.672127 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.672068    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90781578-9f71-4512-8141-d08c8830483b-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-8ws7p\" (UID: \"90781578-9f71-4512-8141-d08c8830483b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8ws7p"
Oct 31 03:24:50.672235 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.672174    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/90781578-9f71-4512-8141-d08c8830483b-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-8ws7p\" (UID: \"90781578-9f71-4512-8141-d08c8830483b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8ws7p"
Oct 31 03:24:50.672358 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.672334    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/90781578-9f71-4512-8141-d08c8830483b-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-8ws7p\" (UID: \"90781578-9f71-4512-8141-d08c8830483b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8ws7p"
Oct 31 03:24:50.672423 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.672389    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90781578-9f71-4512-8141-d08c8830483b-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-8ws7p\" (UID: \"90781578-9f71-4512-8141-d08c8830483b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8ws7p"
Oct 31 03:24:50.673090 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.673072    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/90781578-9f71-4512-8141-d08c8830483b-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-8ws7p\" (UID: \"90781578-9f71-4512-8141-d08c8830483b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8ws7p"
Oct 31 03:24:50.740790 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.740762    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n22kj\" (UniqueName: \"kubernetes.io/projected/90781578-9f71-4512-8141-d08c8830483b-kube-api-access-n22kj\") pod \"kserve-module-controller-manager-7566cf5fc8-8ws7p\" (UID: \"90781578-9f71-4512-8141-d08c8830483b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8ws7p"
Oct 31 03:24:50.796565 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:50.796536    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8ws7p"
Oct 31 03:24:51.267605 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:51.267576    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-8ws7p"]
Oct 31 03:24:51.269456 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:24:51.269426    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90781578_9f71_4512_8141_d08c8830483b.slice/crio-cd0c30fc3d7b412d14de6503a4939a177dff91de140c4986605831046cc14736 WatchSource:0}: Error finding container cd0c30fc3d7b412d14de6503a4939a177dff91de140c4986605831046cc14736: Status 404 returned error can't find the container with id cd0c30fc3d7b412d14de6503a4939a177dff91de140c4986605831046cc14736
Oct 31 03:24:51.284767 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:51.284733    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nnpjm" event={"ID":"290bb433-86e4-4bcb-b609-abb84f34705f","Type":"ContainerStarted","Data":"615a31218d9498ba911f0bdc8ff323ae89eb1dfcdb8dfd3eea9d4ee21c661f3e"}
Oct 31 03:24:51.286877 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:51.286854    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8ws7p" event={"ID":"90781578-9f71-4512-8141-d08c8830483b","Type":"ContainerStarted","Data":"cd0c30fc3d7b412d14de6503a4939a177dff91de140c4986605831046cc14736"}
Oct 31 03:24:55.322649 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:55.322605    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8ws7p" event={"ID":"90781578-9f71-4512-8141-d08c8830483b","Type":"ContainerStarted","Data":"610237a28864d100a585dd2f2b7c879fe720251d7a4214a60cd408b1e5b96b49"}
Oct 31 03:24:55.323112 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:55.322753    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8ws7p"
Oct 31 03:24:55.542950 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:55.542903    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8ws7p" podStartSLOduration=2.351322009 podStartE2EDuration="5.542870523s" totalImagesPullingTime="3.191548514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:55.490836308 +0000 UTC m=+321.001818707" watchObservedRunningTime="2026-10-31 03:24:55.542870523 +0000 UTC m=+321.053852900"
Oct 31 03:24:57.335083 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:57.335043    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nnpjm" event={"ID":"290bb433-86e4-4bcb-b609-abb84f34705f","Type":"ContainerStarted","Data":"0dc8865e86845c77b4bf9ee822d6173caefd39190e2b6011529ceadcb3d88e9b"}
Oct 31 03:24:57.335448 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:57.335138    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nnpjm"
Oct 31 03:24:57.456585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:24:57.456529    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nnpjm" podStartSLOduration=1.8718424219999998 podStartE2EDuration="8.456515234s" totalImagesPullingTime="6.584672812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:57.45485492 +0000 UTC m=+322.965837297" watchObservedRunningTime="2026-10-31 03:24:57.456515234 +0000 UTC m=+322.967497610"
Oct 31 03:25:03.368447 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:03.368410    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nnpjm"
Oct 31 03:25:10.582112 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:10.582074    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-hpwqv"]
Oct 31 03:25:10.596916 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:10.596870    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-hpwqv"
Oct 31 03:25:10.646464 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:10.646429    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-hpwqv"]
Oct 31 03:25:10.693004 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:10.692969    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 31 03:25:10.693242 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:10.693222    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 31 03:25:10.693307 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:10.693241    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 03:25:10.693307 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:10.693247    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2xtxp\""
Oct 31 03:25:10.766652 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:10.766623    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0814729-a524-4d1a-9c18-924142162012-webhook-cert\") pod \"maas-controller-579cf7c6d-hpwqv\" (UID: \"b0814729-a524-4d1a-9c18-924142162012\") " pod="opendatahub/maas-controller-579cf7c6d-hpwqv"
Oct 31 03:25:10.766965 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:10.766941    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvq59\" (UniqueName: \"kubernetes.io/projected/b0814729-a524-4d1a-9c18-924142162012-kube-api-access-vvq59\") pod \"maas-controller-579cf7c6d-hpwqv\" (UID: \"b0814729-a524-4d1a-9c18-924142162012\") " pod="opendatahub/maas-controller-579cf7c6d-hpwqv"
Oct 31 03:25:10.767068 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:10.767053    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0814729-a524-4d1a-9c18-924142162012-metrics-certs\") pod \"maas-controller-579cf7c6d-hpwqv\" (UID: \"b0814729-a524-4d1a-9c18-924142162012\") " pod="opendatahub/maas-controller-579cf7c6d-hpwqv"
Oct 31 03:25:10.868452 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:10.868355    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0814729-a524-4d1a-9c18-924142162012-metrics-certs\") pod \"maas-controller-579cf7c6d-hpwqv\" (UID: \"b0814729-a524-4d1a-9c18-924142162012\") " pod="opendatahub/maas-controller-579cf7c6d-hpwqv"
Oct 31 03:25:10.868452 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:10.868411    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0814729-a524-4d1a-9c18-924142162012-webhook-cert\") pod \"maas-controller-579cf7c6d-hpwqv\" (UID: \"b0814729-a524-4d1a-9c18-924142162012\") " pod="opendatahub/maas-controller-579cf7c6d-hpwqv"
Oct 31 03:25:10.868694 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:10.868464    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvq59\" (UniqueName: \"kubernetes.io/projected/b0814729-a524-4d1a-9c18-924142162012-kube-api-access-vvq59\") pod \"maas-controller-579cf7c6d-hpwqv\" (UID: \"b0814729-a524-4d1a-9c18-924142162012\") " pod="opendatahub/maas-controller-579cf7c6d-hpwqv"
Oct 31 03:25:10.870986 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:10.870961    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0814729-a524-4d1a-9c18-924142162012-metrics-certs\") pod \"maas-controller-579cf7c6d-hpwqv\" (UID: \"b0814729-a524-4d1a-9c18-924142162012\") " pod="opendatahub/maas-controller-579cf7c6d-hpwqv"
Oct 31 03:25:10.871086 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:10.870993    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0814729-a524-4d1a-9c18-924142162012-webhook-cert\") pod \"maas-controller-579cf7c6d-hpwqv\" (UID: \"b0814729-a524-4d1a-9c18-924142162012\") " pod="opendatahub/maas-controller-579cf7c6d-hpwqv"
Oct 31 03:25:10.933381 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:10.933337    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvq59\" (UniqueName: \"kubernetes.io/projected/b0814729-a524-4d1a-9c18-924142162012-kube-api-access-vvq59\") pod \"maas-controller-579cf7c6d-hpwqv\" (UID: \"b0814729-a524-4d1a-9c18-924142162012\") " pod="opendatahub/maas-controller-579cf7c6d-hpwqv"
Oct 31 03:25:11.005701 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:11.005661    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-768d4c8b4f-2kskj"]
Oct 31 03:25:11.008030 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:11.008004    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-768d4c8b4f-2kskj"
Oct 31 03:25:11.070151 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:11.070119    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-768d4c8b4f-2kskj"]
Oct 31 03:25:11.170641 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:11.170558    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e102d26-f8d0-496a-bca4-f2d613e89745-webhook-cert\") pod \"maas-controller-768d4c8b4f-2kskj\" (UID: \"2e102d26-f8d0-496a-bca4-f2d613e89745\") " pod="opendatahub/maas-controller-768d4c8b4f-2kskj"
Oct 31 03:25:11.170641 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:11.170606    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e102d26-f8d0-496a-bca4-f2d613e89745-metrics-certs\") pod \"maas-controller-768d4c8b4f-2kskj\" (UID: \"2e102d26-f8d0-496a-bca4-f2d613e89745\") " pod="opendatahub/maas-controller-768d4c8b4f-2kskj"
Oct 31 03:25:11.170866 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:11.170744    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvrf2\" (UniqueName: \"kubernetes.io/projected/2e102d26-f8d0-496a-bca4-f2d613e89745-kube-api-access-tvrf2\") pod \"maas-controller-768d4c8b4f-2kskj\" (UID: \"2e102d26-f8d0-496a-bca4-f2d613e89745\") " pod="opendatahub/maas-controller-768d4c8b4f-2kskj"
Oct 31 03:25:11.211059 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:11.211024    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-hpwqv"
Oct 31 03:25:11.272585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:11.272547    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e102d26-f8d0-496a-bca4-f2d613e89745-webhook-cert\") pod \"maas-controller-768d4c8b4f-2kskj\" (UID: \"2e102d26-f8d0-496a-bca4-f2d613e89745\") " pod="opendatahub/maas-controller-768d4c8b4f-2kskj"
Oct 31 03:25:11.272678 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:11.272632    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e102d26-f8d0-496a-bca4-f2d613e89745-metrics-certs\") pod \"maas-controller-768d4c8b4f-2kskj\" (UID: \"2e102d26-f8d0-496a-bca4-f2d613e89745\") " pod="opendatahub/maas-controller-768d4c8b4f-2kskj"
Oct 31 03:25:11.272734 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:11.272697    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvrf2\" (UniqueName: \"kubernetes.io/projected/2e102d26-f8d0-496a-bca4-f2d613e89745-kube-api-access-tvrf2\") pod \"maas-controller-768d4c8b4f-2kskj\" (UID: \"2e102d26-f8d0-496a-bca4-f2d613e89745\") " pod="opendatahub/maas-controller-768d4c8b4f-2kskj"
Oct 31 03:25:11.278479 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:11.277941    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e102d26-f8d0-496a-bca4-f2d613e89745-webhook-cert\") pod \"maas-controller-768d4c8b4f-2kskj\" (UID: \"2e102d26-f8d0-496a-bca4-f2d613e89745\") " pod="opendatahub/maas-controller-768d4c8b4f-2kskj"
Oct 31 03:25:11.279412 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:11.279367    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e102d26-f8d0-496a-bca4-f2d613e89745-metrics-certs\") pod \"maas-controller-768d4c8b4f-2kskj\" (UID: \"2e102d26-f8d0-496a-bca4-f2d613e89745\") " pod="opendatahub/maas-controller-768d4c8b4f-2kskj"
Oct 31 03:25:11.334445 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:11.334415    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvrf2\" (UniqueName: \"kubernetes.io/projected/2e102d26-f8d0-496a-bca4-f2d613e89745-kube-api-access-tvrf2\") pod \"maas-controller-768d4c8b4f-2kskj\" (UID: \"2e102d26-f8d0-496a-bca4-f2d613e89745\") " pod="opendatahub/maas-controller-768d4c8b4f-2kskj"
Oct 31 03:25:11.619977 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:11.619942    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-768d4c8b4f-2kskj"
Oct 31 03:25:11.625924 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:11.625903    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-hpwqv"]
Oct 31 03:25:11.629184 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:25:11.629160    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0814729_a524_4d1a_9c18_924142162012.slice/crio-7b9f4811b340d2c642d9a7102b08bd9fdfe7ef4ce6ed5cc7225ea81234a6bb4b WatchSource:0}: Error finding container 7b9f4811b340d2c642d9a7102b08bd9fdfe7ef4ce6ed5cc7225ea81234a6bb4b: Status 404 returned error can't find the container with id 7b9f4811b340d2c642d9a7102b08bd9fdfe7ef4ce6ed5cc7225ea81234a6bb4b
Oct 31 03:25:11.994859 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:11.994827    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-768d4c8b4f-2kskj"]
Oct 31 03:25:11.996253 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:25:11.996232    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e102d26_f8d0_496a_bca4_f2d613e89745.slice/crio-fe8a067f25e3a9d8a097357502e331b06f6a7003121ed6b651fe702907d2a585 WatchSource:0}: Error finding container fe8a067f25e3a9d8a097357502e331b06f6a7003121ed6b651fe702907d2a585: Status 404 returned error can't find the container with id fe8a067f25e3a9d8a097357502e331b06f6a7003121ed6b651fe702907d2a585
Oct 31 03:25:12.423312 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:12.423230    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-hpwqv" event={"ID":"b0814729-a524-4d1a-9c18-924142162012","Type":"ContainerStarted","Data":"7b9f4811b340d2c642d9a7102b08bd9fdfe7ef4ce6ed5cc7225ea81234a6bb4b"}
Oct 31 03:25:12.424478 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:12.424449    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-768d4c8b4f-2kskj" event={"ID":"2e102d26-f8d0-496a-bca4-f2d613e89745","Type":"ContainerStarted","Data":"fe8a067f25e3a9d8a097357502e331b06f6a7003121ed6b651fe702907d2a585"}
Oct 31 03:25:16.330022 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:16.329990    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8ws7p"
Oct 31 03:25:16.449601 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:16.449563    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-768d4c8b4f-2kskj" event={"ID":"2e102d26-f8d0-496a-bca4-f2d613e89745","Type":"ContainerStarted","Data":"4780a92139ba74c5277f67d2e35158a3314096b4f644e7897d7bf4ab062c527a"}
Oct 31 03:25:16.449787 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:16.449631    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-768d4c8b4f-2kskj"
Oct 31 03:25:16.451230 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:16.451204    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-hpwqv" event={"ID":"b0814729-a524-4d1a-9c18-924142162012","Type":"ContainerStarted","Data":"538377b1835e6ca654d2a7d7381cbe516d906347fa388cd8b9562c2b6e8e3d8c"}
Oct 31 03:25:16.451345 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:16.451253    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-hpwqv"
Oct 31 03:25:16.620045 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:16.619953    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-hpwqv" podStartSLOduration=1.917729821 podStartE2EDuration="6.619939736s" totalImagesPullingTime="4.702209915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:16.617721253 +0000 UTC m=+342.128703632" watchObservedRunningTime="2026-10-31 03:25:16.619939736 +0000 UTC m=+342.130922113"
Oct 31 03:25:16.742383 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:16.742334    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-768d4c8b4f-2kskj" podStartSLOduration=2.397723126 podStartE2EDuration="6.742317954s" totalImagesPullingTime="4.344594828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:16.740229796 +0000 UTC m=+342.251212185" watchObservedRunningTime="2026-10-31 03:25:16.742317954 +0000 UTC m=+342.253300339"
Oct 31 03:25:19.000946 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:25:19.000911    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:25:19.833713 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:19.833677    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5brx"]
Oct 31 03:25:19.837472 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:19.837451    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5brx"
Oct 31 03:25:19.861388 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:19.861350    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/23ab1790-f42f-4bad-a72f-49800dee980c-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5brx\" (UID: \"23ab1790-f42f-4bad-a72f-49800dee980c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5brx"
Oct 31 03:25:19.861537 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:19.861462    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/23ab1790-f42f-4bad-a72f-49800dee980c-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5brx\" (UID: \"23ab1790-f42f-4bad-a72f-49800dee980c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5brx"
Oct 31 03:25:19.861537 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:19.861505    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j79v\" (UniqueName: \"kubernetes.io/projected/23ab1790-f42f-4bad-a72f-49800dee980c-kube-api-access-2j79v\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5brx\" (UID: \"23ab1790-f42f-4bad-a72f-49800dee980c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5brx"
Oct 31 03:25:19.861611 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:19.861594    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23ab1790-f42f-4bad-a72f-49800dee980c-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5brx\" (UID: \"23ab1790-f42f-4bad-a72f-49800dee980c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5brx"
Oct 31 03:25:19.890154 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:19.890127    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 31 03:25:19.940013 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:19.939973    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 31 03:25:19.940013 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:19.939976    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 31 03:25:19.940239 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:19.940066    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-2glx7\""
Oct 31 03:25:19.942673 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:19.942655    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5brx"]
Oct 31 03:25:19.962636 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:19.962608    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/23ab1790-f42f-4bad-a72f-49800dee980c-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5brx\" (UID: \"23ab1790-f42f-4bad-a72f-49800dee980c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5brx"
Oct 31 03:25:19.962817 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:19.962658    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j79v\" (UniqueName: \"kubernetes.io/projected/23ab1790-f42f-4bad-a72f-49800dee980c-kube-api-access-2j79v\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5brx\" (UID: \"23ab1790-f42f-4bad-a72f-49800dee980c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5brx"
Oct 31 03:25:19.962817 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:19.962688    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23ab1790-f42f-4bad-a72f-49800dee980c-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5brx\" (UID: \"23ab1790-f42f-4bad-a72f-49800dee980c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5brx"
Oct 31 03:25:19.962817 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:19.962713    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/23ab1790-f42f-4bad-a72f-49800dee980c-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5brx\" (UID: \"23ab1790-f42f-4bad-a72f-49800dee980c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5brx"
Oct 31 03:25:19.962817 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:25:19.962755    4447 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Oct 31 03:25:19.963029 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:25:19.962835    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23ab1790-f42f-4bad-a72f-49800dee980c-proxy-source podName:23ab1790-f42f-4bad-a72f-49800dee980c nodeName:}" failed. No retries permitted until 2026-10-31 03:25:20.46281158 +0000 UTC m=+345.973793960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/23ab1790-f42f-4bad-a72f-49800dee980c-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-q5brx" (UID: "23ab1790-f42f-4bad-a72f-49800dee980c") : configmap "usage-tenancy-proxy-source" not found
Oct 31 03:25:19.963029 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:25:19.962849    4447 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Oct 31 03:25:19.963029 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:25:19.962936    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23ab1790-f42f-4bad-a72f-49800dee980c-tls-cert podName:23ab1790-f42f-4bad-a72f-49800dee980c nodeName:}" failed. No retries permitted until 2026-10-31 03:25:20.462919168 +0000 UTC m=+345.973901536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/23ab1790-f42f-4bad-a72f-49800dee980c-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-q5brx" (UID: "23ab1790-f42f-4bad-a72f-49800dee980c") : secret "usage-logs-tenancy-proxy-tls" not found
Oct 31 03:25:19.963363 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:19.963347    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23ab1790-f42f-4bad-a72f-49800dee980c-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5brx\" (UID: \"23ab1790-f42f-4bad-a72f-49800dee980c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5brx"
Oct 31 03:25:20.025522 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:20.025484    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j79v\" (UniqueName: \"kubernetes.io/projected/23ab1790-f42f-4bad-a72f-49800dee980c-kube-api-access-2j79v\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5brx\" (UID: \"23ab1790-f42f-4bad-a72f-49800dee980c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5brx"
Oct 31 03:25:20.468188 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:20.468153    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/23ab1790-f42f-4bad-a72f-49800dee980c-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5brx\" (UID: \"23ab1790-f42f-4bad-a72f-49800dee980c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5brx"
Oct 31 03:25:20.468388 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:20.468277    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/23ab1790-f42f-4bad-a72f-49800dee980c-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5brx\" (UID: \"23ab1790-f42f-4bad-a72f-49800dee980c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5brx"
Oct 31 03:25:20.469166 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:20.469141    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/23ab1790-f42f-4bad-a72f-49800dee980c-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5brx\" (UID: \"23ab1790-f42f-4bad-a72f-49800dee980c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5brx"
Oct 31 03:25:20.471107 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:20.471083    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/23ab1790-f42f-4bad-a72f-49800dee980c-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q5brx\" (UID: \"23ab1790-f42f-4bad-a72f-49800dee980c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5brx"
Oct 31 03:25:20.749742 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:20.749654    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5brx"
Oct 31 03:25:21.128602 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:21.128579    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5brx"]
Oct 31 03:25:21.130325 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:25:21.130290    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ab1790_f42f_4bad_a72f_49800dee980c.slice/crio-6358d006a8a19c745412d63472824f29a786aadad971d4bdcb19574d015d2a16 WatchSource:0}: Error finding container 6358d006a8a19c745412d63472824f29a786aadad971d4bdcb19574d015d2a16: Status 404 returned error can't find the container with id 6358d006a8a19c745412d63472824f29a786aadad971d4bdcb19574d015d2a16
Oct 31 03:25:21.479435 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:21.479400    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5brx" event={"ID":"23ab1790-f42f-4bad-a72f-49800dee980c","Type":"ContainerStarted","Data":"6358d006a8a19c745412d63472824f29a786aadad971d4bdcb19574d015d2a16"}
Oct 31 03:25:27.460256 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:27.460217    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-hpwqv"
Oct 31 03:25:27.460847 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:27.460826    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-768d4c8b4f-2kskj"
Oct 31 03:25:27.734271 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:27.734176    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-hpwqv"]
Oct 31 03:25:27.734518 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:27.734374    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-hpwqv" podUID="b0814729-a524-4d1a-9c18-924142162012" containerName="manager" containerID="cri-o://538377b1835e6ca654d2a7d7381cbe516d906347fa388cd8b9562c2b6e8e3d8c" gracePeriod=30
Oct 31 03:25:28.081641 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:28.081300    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-hpwqv"
Oct 31 03:25:28.153135 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:28.153102    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0814729-a524-4d1a-9c18-924142162012-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0814729-a524-4d1a-9c18-924142162012-metrics-certs\") pod \"b0814729-a524-4d1a-9c18-924142162012\" (UID: \"b0814729-a524-4d1a-9c18-924142162012\") "
Oct 31 03:25:28.153311 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:28.153245    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0814729-a524-4d1a-9c18-924142162012-webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0814729-a524-4d1a-9c18-924142162012-webhook-cert\") pod \"b0814729-a524-4d1a-9c18-924142162012\" (UID: \"b0814729-a524-4d1a-9c18-924142162012\") "
Oct 31 03:25:28.153311 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:28.153284    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0814729-a524-4d1a-9c18-924142162012-kube-api-access-vvq59\" (UniqueName: \"kubernetes.io/projected/b0814729-a524-4d1a-9c18-924142162012-kube-api-access-vvq59\") pod \"b0814729-a524-4d1a-9c18-924142162012\" (UID: \"b0814729-a524-4d1a-9c18-924142162012\") "
Oct 31 03:25:28.155860 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:28.155827    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0814729-a524-4d1a-9c18-924142162012-metrics-certs" pod "b0814729-a524-4d1a-9c18-924142162012" (UID: "b0814729-a524-4d1a-9c18-924142162012"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:25:28.155993 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:28.155855    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0814729-a524-4d1a-9c18-924142162012-webhook-cert" pod "b0814729-a524-4d1a-9c18-924142162012" (UID: "b0814729-a524-4d1a-9c18-924142162012"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:25:28.156869 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:28.156844    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0814729-a524-4d1a-9c18-924142162012-kube-api-access-vvq59" pod "b0814729-a524-4d1a-9c18-924142162012" (UID: "b0814729-a524-4d1a-9c18-924142162012"). InnerVolumeSpecName "kube-api-access-vvq59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:25:28.254625 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:28.254590    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0814729-a524-4d1a-9c18-924142162012-metrics-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:25:28.254625 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:28.254629    4447 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0814729-a524-4d1a-9c18-924142162012-webhook-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:25:28.254912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:28.254644    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvq59\" (UniqueName: \"kubernetes.io/projected/b0814729-a524-4d1a-9c18-924142162012-kube-api-access-vvq59\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:25:28.523453 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:28.523407    4447 generic.go:356] "Generic (PLEG): container finished" podID="b0814729-a524-4d1a-9c18-924142162012" containerID="538377b1835e6ca654d2a7d7381cbe516d906347fa388cd8b9562c2b6e8e3d8c" exitCode=0
Oct 31 03:25:28.523945 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:28.523505    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-hpwqv" event={"ID":"b0814729-a524-4d1a-9c18-924142162012","Type":"ContainerDied","Data":"538377b1835e6ca654d2a7d7381cbe516d906347fa388cd8b9562c2b6e8e3d8c"}
Oct 31 03:25:28.523945 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:28.523587    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-hpwqv" event={"ID":"b0814729-a524-4d1a-9c18-924142162012","Type":"ContainerDied","Data":"7b9f4811b340d2c642d9a7102b08bd9fdfe7ef4ce6ed5cc7225ea81234a6bb4b"}
Oct 31 03:25:28.523945 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:28.523620    4447 scope.go:122] "RemoveContainer" containerID="538377b1835e6ca654d2a7d7381cbe516d906347fa388cd8b9562c2b6e8e3d8c"
Oct 31 03:25:28.523945 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:28.523779    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-hpwqv"
Oct 31 03:25:28.544592 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:28.544505    4447 scope.go:122] "RemoveContainer" containerID="538377b1835e6ca654d2a7d7381cbe516d906347fa388cd8b9562c2b6e8e3d8c"
Oct 31 03:25:28.544836 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:25:28.544813    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"538377b1835e6ca654d2a7d7381cbe516d906347fa388cd8b9562c2b6e8e3d8c\": container with ID starting with 538377b1835e6ca654d2a7d7381cbe516d906347fa388cd8b9562c2b6e8e3d8c not found: ID does not exist" containerID="538377b1835e6ca654d2a7d7381cbe516d906347fa388cd8b9562c2b6e8e3d8c"
Oct 31 03:25:28.544963 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:28.544848    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538377b1835e6ca654d2a7d7381cbe516d906347fa388cd8b9562c2b6e8e3d8c"} err="failed to get container status \"538377b1835e6ca654d2a7d7381cbe516d906347fa388cd8b9562c2b6e8e3d8c\": rpc error: code = NotFound desc = could not find container \"538377b1835e6ca654d2a7d7381cbe516d906347fa388cd8b9562c2b6e8e3d8c\": container with ID starting with 538377b1835e6ca654d2a7d7381cbe516d906347fa388cd8b9562c2b6e8e3d8c not found: ID does not exist"
Oct 31 03:25:28.724442 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:28.724411    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-hpwqv"]
Oct 31 03:25:28.730760 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:28.730725    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-hpwqv"]
Oct 31 03:25:30.610067 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:30.609669    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0814729-a524-4d1a-9c18-924142162012" path="/var/lib/kubelet/pods/b0814729-a524-4d1a-9c18-924142162012/volumes"
Oct 31 03:25:36.572570 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:36.572475    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5brx" event={"ID":"23ab1790-f42f-4bad-a72f-49800dee980c","Type":"ContainerStarted","Data":"706de955da287b419fb3fbfd22288791ff9abc21f6eaf5d1c17f13e757cc3ec9"}
Oct 31 03:25:36.572940 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:36.572666    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5brx"
Oct 31 03:25:36.690968 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:36.690910    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5brx" podStartSLOduration=2.533731806 podStartE2EDuration="17.690867758s" totalImagesPullingTime="15.157135952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:36.69014145 +0000 UTC m=+362.201123825" watchObservedRunningTime="2026-10-31 03:25:36.690867758 +0000 UTC m=+362.201850134"
Oct 31 03:25:47.583840 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:47.583805    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q5brx"
Oct 31 03:25:49.005162 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:25:49.005125    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:25:53.444960 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:53.442941    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl"]
Oct 31 03:25:53.444960 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:53.443853    4447 cpu_manager.go:414] "containerMap: removing container" podUID="b0814729-a524-4d1a-9c18-924142162012" containerName="manager"
Oct 31 03:25:53.444960 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:53.443871    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0814729-a524-4d1a-9c18-924142162012" containerName="manager"
Oct 31 03:25:53.444960 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:53.444189    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0814729-a524-4d1a-9c18-924142162012" containerName="manager"
Oct 31 03:25:53.452437 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:53.452411    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl"
Oct 31 03:25:53.505700 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:53.505670    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl"]
Oct 31 03:25:53.516544 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:53.516519    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e0f40c-fb0c-4941-ba96-13f44169a644-openshift-service-ca\") pod \"maas-api-6d669c7fcd-fhkjl\" (UID: \"a0e0f40c-fb0c-4941-ba96-13f44169a644\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl"
Oct 31 03:25:53.516688 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:53.516571    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0e0f40c-fb0c-4941-ba96-13f44169a644-metrics-certs\") pod \"maas-api-6d669c7fcd-fhkjl\" (UID: \"a0e0f40c-fb0c-4941-ba96-13f44169a644\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl"
Oct 31 03:25:53.516688 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:53.516591    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0e0f40c-fb0c-4941-ba96-13f44169a644-maas-api-tls\") pod \"maas-api-6d669c7fcd-fhkjl\" (UID: \"a0e0f40c-fb0c-4941-ba96-13f44169a644\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl"
Oct 31 03:25:53.516778 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:53.516711    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdfxt\" (UniqueName: \"kubernetes.io/projected/a0e0f40c-fb0c-4941-ba96-13f44169a644-kube-api-access-hdfxt\") pod \"maas-api-6d669c7fcd-fhkjl\" (UID: \"a0e0f40c-fb0c-4941-ba96-13f44169a644\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl"
Oct 31 03:25:53.552873 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:53.552833    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-spbn9\""
Oct 31 03:25:53.553074 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:53.552950    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 03:25:53.557090 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:53.557065    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 03:25:53.618041 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:53.618002    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e0f40c-fb0c-4941-ba96-13f44169a644-openshift-service-ca\") pod \"maas-api-6d669c7fcd-fhkjl\" (UID: \"a0e0f40c-fb0c-4941-ba96-13f44169a644\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl"
Oct 31 03:25:53.618234 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:53.618134    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0e0f40c-fb0c-4941-ba96-13f44169a644-metrics-certs\") pod \"maas-api-6d669c7fcd-fhkjl\" (UID: \"a0e0f40c-fb0c-4941-ba96-13f44169a644\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl"
Oct 31 03:25:53.618234 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:53.618165    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0e0f40c-fb0c-4941-ba96-13f44169a644-maas-api-tls\") pod \"maas-api-6d669c7fcd-fhkjl\" (UID: \"a0e0f40c-fb0c-4941-ba96-13f44169a644\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl"
Oct 31 03:25:53.618234 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:53.618225    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdfxt\" (UniqueName: \"kubernetes.io/projected/a0e0f40c-fb0c-4941-ba96-13f44169a644-kube-api-access-hdfxt\") pod \"maas-api-6d669c7fcd-fhkjl\" (UID: \"a0e0f40c-fb0c-4941-ba96-13f44169a644\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl"
Oct 31 03:25:53.618801 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:53.618779    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e0f40c-fb0c-4941-ba96-13f44169a644-openshift-service-ca\") pod \"maas-api-6d669c7fcd-fhkjl\" (UID: \"a0e0f40c-fb0c-4941-ba96-13f44169a644\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl"
Oct 31 03:25:53.620946 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:53.620871    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0e0f40c-fb0c-4941-ba96-13f44169a644-maas-api-tls\") pod \"maas-api-6d669c7fcd-fhkjl\" (UID: \"a0e0f40c-fb0c-4941-ba96-13f44169a644\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl"
Oct 31 03:25:53.621055 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:53.620956    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0e0f40c-fb0c-4941-ba96-13f44169a644-metrics-certs\") pod \"maas-api-6d669c7fcd-fhkjl\" (UID: \"a0e0f40c-fb0c-4941-ba96-13f44169a644\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl"
Oct 31 03:25:53.680352 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:53.680315    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdfxt\" (UniqueName: \"kubernetes.io/projected/a0e0f40c-fb0c-4941-ba96-13f44169a644-kube-api-access-hdfxt\") pod \"maas-api-6d669c7fcd-fhkjl\" (UID: \"a0e0f40c-fb0c-4941-ba96-13f44169a644\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl"
Oct 31 03:25:53.766380 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:53.766345    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl"
Oct 31 03:25:54.149595 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:54.149572    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl"]
Oct 31 03:25:54.151438 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:25:54.151412    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e0f40c_fb0c_4941_ba96_13f44169a644.slice/crio-38a36f6b521681a870005ead7f189dbd12e70a3fad99dc19774fbd64faec351c WatchSource:0}: Error finding container 38a36f6b521681a870005ead7f189dbd12e70a3fad99dc19774fbd64faec351c: Status 404 returned error can't find the container with id 38a36f6b521681a870005ead7f189dbd12e70a3fad99dc19774fbd64faec351c
Oct 31 03:25:54.670105 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:54.670071    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl" event={"ID":"a0e0f40c-fb0c-4941-ba96-13f44169a644","Type":"ContainerStarted","Data":"38a36f6b521681a870005ead7f189dbd12e70a3fad99dc19774fbd64faec351c"}
Oct 31 03:25:56.682354 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:56.682313    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl" event={"ID":"a0e0f40c-fb0c-4941-ba96-13f44169a644","Type":"ContainerStarted","Data":"29e5c3d6d2012ffad0a756c4e0b9d8a9c4b1b14e062a5b3bbc330f5ad0fd5976"}
Oct 31 03:25:56.682746 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:56.682390    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl"
Oct 31 03:25:56.800949 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:56.800861    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl" podStartSLOduration=1.9037068019999999 podStartE2EDuration="3.800840443s" totalImagesPullingTime="1.897133641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:56.800478742 +0000 UTC m=+382.311461157" watchObservedRunningTime="2026-10-31 03:25:56.800840443 +0000 UTC m=+382.311822821"
Oct 31 03:25:59.513272 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.513238    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-cbh4k"]
Oct 31 03:25:59.517041 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.517022    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-cbh4k"
Oct 31 03:25:59.576515 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.576482    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-cbh4k"]
Oct 31 03:25:59.582212 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.582183    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgv6p\" (UniqueName: \"kubernetes.io/projected/da0b818f-0cfd-4c31-a808-588b3138c91b-kube-api-access-wgv6p\") pod \"kserve-controller-manager-776cc66d47-cbh4k\" (UID: \"da0b818f-0cfd-4c31-a808-588b3138c91b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-cbh4k"
Oct 31 03:25:59.582362 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.582308    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da0b818f-0cfd-4c31-a808-588b3138c91b-cert\") pod \"kserve-controller-manager-776cc66d47-cbh4k\" (UID: \"da0b818f-0cfd-4c31-a808-588b3138c91b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-cbh4k"
Oct 31 03:25:59.622352 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.622322    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 31 03:25:59.622564 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.622548    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-dvmzz\""
Oct 31 03:25:59.683916 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.683854    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgv6p\" (UniqueName: \"kubernetes.io/projected/da0b818f-0cfd-4c31-a808-588b3138c91b-kube-api-access-wgv6p\") pod \"kserve-controller-manager-776cc66d47-cbh4k\" (UID: \"da0b818f-0cfd-4c31-a808-588b3138c91b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-cbh4k"
Oct 31 03:25:59.684237 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.684207    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da0b818f-0cfd-4c31-a808-588b3138c91b-cert\") pod \"kserve-controller-manager-776cc66d47-cbh4k\" (UID: \"da0b818f-0cfd-4c31-a808-588b3138c91b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-cbh4k"
Oct 31 03:25:59.686674 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.686650    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da0b818f-0cfd-4c31-a808-588b3138c91b-cert\") pod \"kserve-controller-manager-776cc66d47-cbh4k\" (UID: \"da0b818f-0cfd-4c31-a808-588b3138c91b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-cbh4k"
Oct 31 03:25:59.748164 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.748134    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgv6p\" (UniqueName: \"kubernetes.io/projected/da0b818f-0cfd-4c31-a808-588b3138c91b-kube-api-access-wgv6p\") pod \"kserve-controller-manager-776cc66d47-cbh4k\" (UID: \"da0b818f-0cfd-4c31-a808-588b3138c91b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-cbh4k"
Oct 31 03:25:59.763323 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.763251    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-vvkxg"]
Oct 31 03:25:59.766844 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.766819    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-vvkxg"
Oct 31 03:25:59.828271 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.828228    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-vvkxg"]
Oct 31 03:25:59.829879 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.829846    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-cbh4k"
Oct 31 03:25:59.848699 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.848659    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-9gvtt\""
Oct 31 03:25:59.848699 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.848670    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 31 03:25:59.896684 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.894387    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9241326-bbbe-4a12-aabd-67e56ee8e55e-tls-certs\") pod \"model-serving-api-75bd89fdfc-vvkxg\" (UID: \"c9241326-bbbe-4a12-aabd-67e56ee8e55e\") " pod="opendatahub/model-serving-api-75bd89fdfc-vvkxg"
Oct 31 03:25:59.896684 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.894536    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdpd2\" (UniqueName: \"kubernetes.io/projected/c9241326-bbbe-4a12-aabd-67e56ee8e55e-kube-api-access-fdpd2\") pod \"model-serving-api-75bd89fdfc-vvkxg\" (UID: \"c9241326-bbbe-4a12-aabd-67e56ee8e55e\") " pod="opendatahub/model-serving-api-75bd89fdfc-vvkxg"
Oct 31 03:25:59.899283 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.897037    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-fh6m6"]
Oct 31 03:25:59.905730 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.905696    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-fh6m6"
Oct 31 03:25:59.947608 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.947564    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-fh6m6"]
Oct 31 03:25:59.959030 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.958996    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-qvxhs\""
Oct 31 03:25:59.959203 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.959052    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 31 03:25:59.959203 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.959051    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 31 03:25:59.998914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.998384    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdpd2\" (UniqueName: \"kubernetes.io/projected/c9241326-bbbe-4a12-aabd-67e56ee8e55e-kube-api-access-fdpd2\") pod \"model-serving-api-75bd89fdfc-vvkxg\" (UID: \"c9241326-bbbe-4a12-aabd-67e56ee8e55e\") " pod="opendatahub/model-serving-api-75bd89fdfc-vvkxg"
Oct 31 03:25:59.998914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.998466    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9241326-bbbe-4a12-aabd-67e56ee8e55e-tls-certs\") pod \"model-serving-api-75bd89fdfc-vvkxg\" (UID: \"c9241326-bbbe-4a12-aabd-67e56ee8e55e\") " pod="opendatahub/model-serving-api-75bd89fdfc-vvkxg"
Oct 31 03:25:59.998914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.998500    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73b2caea-8bec-45ab-ad59-0415870d7b05-metrics-certs\") pod \"odh-model-controller-769875db56-fh6m6\" (UID: \"73b2caea-8bec-45ab-ad59-0415870d7b05\") " pod="opendatahub/odh-model-controller-769875db56-fh6m6"
Oct 31 03:25:59.998914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.998569    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73b2caea-8bec-45ab-ad59-0415870d7b05-cert\") pod \"odh-model-controller-769875db56-fh6m6\" (UID: \"73b2caea-8bec-45ab-ad59-0415870d7b05\") " pod="opendatahub/odh-model-controller-769875db56-fh6m6"
Oct 31 03:25:59.998914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:25:59.998604    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxn44\" (UniqueName: \"kubernetes.io/projected/73b2caea-8bec-45ab-ad59-0415870d7b05-kube-api-access-wxn44\") pod \"odh-model-controller-769875db56-fh6m6\" (UID: \"73b2caea-8bec-45ab-ad59-0415870d7b05\") " pod="opendatahub/odh-model-controller-769875db56-fh6m6"
Oct 31 03:26:00.001770 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:00.001740    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9241326-bbbe-4a12-aabd-67e56ee8e55e-tls-certs\") pod \"model-serving-api-75bd89fdfc-vvkxg\" (UID: \"c9241326-bbbe-4a12-aabd-67e56ee8e55e\") " pod="opendatahub/model-serving-api-75bd89fdfc-vvkxg"
Oct 31 03:26:00.060263 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:00.060194    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdpd2\" (UniqueName: \"kubernetes.io/projected/c9241326-bbbe-4a12-aabd-67e56ee8e55e-kube-api-access-fdpd2\") pod \"model-serving-api-75bd89fdfc-vvkxg\" (UID: \"c9241326-bbbe-4a12-aabd-67e56ee8e55e\") " pod="opendatahub/model-serving-api-75bd89fdfc-vvkxg"
Oct 31 03:26:00.097392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:00.097086    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-vvkxg"
Oct 31 03:26:00.099210 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:00.099174    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73b2caea-8bec-45ab-ad59-0415870d7b05-cert\") pod \"odh-model-controller-769875db56-fh6m6\" (UID: \"73b2caea-8bec-45ab-ad59-0415870d7b05\") " pod="opendatahub/odh-model-controller-769875db56-fh6m6"
Oct 31 03:26:00.099378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:00.099226    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxn44\" (UniqueName: \"kubernetes.io/projected/73b2caea-8bec-45ab-ad59-0415870d7b05-kube-api-access-wxn44\") pod \"odh-model-controller-769875db56-fh6m6\" (UID: \"73b2caea-8bec-45ab-ad59-0415870d7b05\") " pod="opendatahub/odh-model-controller-769875db56-fh6m6"
Oct 31 03:26:00.099378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:00.099330    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73b2caea-8bec-45ab-ad59-0415870d7b05-metrics-certs\") pod \"odh-model-controller-769875db56-fh6m6\" (UID: \"73b2caea-8bec-45ab-ad59-0415870d7b05\") " pod="opendatahub/odh-model-controller-769875db56-fh6m6"
Oct 31 03:26:00.101902 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:00.101865    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73b2caea-8bec-45ab-ad59-0415870d7b05-cert\") pod \"odh-model-controller-769875db56-fh6m6\" (UID: \"73b2caea-8bec-45ab-ad59-0415870d7b05\") " pod="opendatahub/odh-model-controller-769875db56-fh6m6"
Oct 31 03:26:00.102042 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:00.102019    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73b2caea-8bec-45ab-ad59-0415870d7b05-metrics-certs\") pod \"odh-model-controller-769875db56-fh6m6\" (UID: \"73b2caea-8bec-45ab-ad59-0415870d7b05\") " pod="opendatahub/odh-model-controller-769875db56-fh6m6"
Oct 31 03:26:00.161588 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:00.161556    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxn44\" (UniqueName: \"kubernetes.io/projected/73b2caea-8bec-45ab-ad59-0415870d7b05-kube-api-access-wxn44\") pod \"odh-model-controller-769875db56-fh6m6\" (UID: \"73b2caea-8bec-45ab-ad59-0415870d7b05\") " pod="opendatahub/odh-model-controller-769875db56-fh6m6"
Oct 31 03:26:00.231338 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:00.231277    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-fh6m6"
Oct 31 03:26:00.232720 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:00.232693    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-cbh4k"]
Oct 31 03:26:00.234010 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:26:00.233983    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0b818f_0cfd_4c31_a808_588b3138c91b.slice/crio-2f4bffa5e61ae14becb4c5fa5413ae3656ea6ea9ad9348eff86ed5f9642e2086 WatchSource:0}: Error finding container 2f4bffa5e61ae14becb4c5fa5413ae3656ea6ea9ad9348eff86ed5f9642e2086: Status 404 returned error can't find the container with id 2f4bffa5e61ae14becb4c5fa5413ae3656ea6ea9ad9348eff86ed5f9642e2086
Oct 31 03:26:00.469661 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:00.469632    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-vvkxg"]
Oct 31 03:26:00.471286 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:26:00.471255    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9241326_bbbe_4a12_aabd_67e56ee8e55e.slice/crio-d75e0dde4490f8f1332d298ce3536e2136ffaa70a8dc59566ac386c508e924ed WatchSource:0}: Error finding container d75e0dde4490f8f1332d298ce3536e2136ffaa70a8dc59566ac386c508e924ed: Status 404 returned error can't find the container with id d75e0dde4490f8f1332d298ce3536e2136ffaa70a8dc59566ac386c508e924ed
Oct 31 03:26:00.626155 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:00.626065    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-fh6m6"]
Oct 31 03:26:00.629615 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:26:00.629592    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b2caea_8bec_45ab_ad59_0415870d7b05.slice/crio-b75ed69222807af879a70e77b94abd1c720d459809431a9eef8f75471bb9be50 WatchSource:0}: Error finding container b75ed69222807af879a70e77b94abd1c720d459809431a9eef8f75471bb9be50: Status 404 returned error can't find the container with id b75ed69222807af879a70e77b94abd1c720d459809431a9eef8f75471bb9be50
Oct 31 03:26:00.705594 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:00.705560    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-vvkxg" event={"ID":"c9241326-bbbe-4a12-aabd-67e56ee8e55e","Type":"ContainerStarted","Data":"d75e0dde4490f8f1332d298ce3536e2136ffaa70a8dc59566ac386c508e924ed"}
Oct 31 03:26:00.706895 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:00.706856    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-cbh4k" event={"ID":"da0b818f-0cfd-4c31-a808-588b3138c91b","Type":"ContainerStarted","Data":"2f4bffa5e61ae14becb4c5fa5413ae3656ea6ea9ad9348eff86ed5f9642e2086"}
Oct 31 03:26:00.708442 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:00.708418    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-fh6m6" event={"ID":"73b2caea-8bec-45ab-ad59-0415870d7b05","Type":"ContainerStarted","Data":"b75ed69222807af879a70e77b94abd1c720d459809431a9eef8f75471bb9be50"}
Oct 31 03:26:02.706785 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:02.706324    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl"
Oct 31 03:26:03.641390 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:03.640196    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-szbkl"]
Oct 31 03:26:03.709499 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:03.709269    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-szbkl"]
Oct 31 03:26:03.709499 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:03.709408    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-szbkl"
Oct 31 03:26:03.744359 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:03.744285    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2t4g\" (UniqueName: \"kubernetes.io/projected/74ca5223-2deb-45ed-af0e-422b1cb677d2-kube-api-access-j2t4g\") pod \"authorino-6f6944bcdd-szbkl\" (UID: \"74ca5223-2deb-45ed-af0e-422b1cb677d2\") " pod="kuadrant-system/authorino-6f6944bcdd-szbkl"
Oct 31 03:26:03.845594 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:03.845558    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2t4g\" (UniqueName: \"kubernetes.io/projected/74ca5223-2deb-45ed-af0e-422b1cb677d2-kube-api-access-j2t4g\") pod \"authorino-6f6944bcdd-szbkl\" (UID: \"74ca5223-2deb-45ed-af0e-422b1cb677d2\") " pod="kuadrant-system/authorino-6f6944bcdd-szbkl"
Oct 31 03:26:03.908005 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:03.907912    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2t4g\" (UniqueName: \"kubernetes.io/projected/74ca5223-2deb-45ed-af0e-422b1cb677d2-kube-api-access-j2t4g\") pod \"authorino-6f6944bcdd-szbkl\" (UID: \"74ca5223-2deb-45ed-af0e-422b1cb677d2\") " pod="kuadrant-system/authorino-6f6944bcdd-szbkl"
Oct 31 03:26:04.040208 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:04.040172    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-szbkl"
Oct 31 03:26:04.049364 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:04.049330    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-szbkl"]
Oct 31 03:26:04.073513 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:04.073426    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68bcb74d46-gzvzp"]
Oct 31 03:26:04.128982 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:04.128917    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68bcb74d46-gzvzp"
Oct 31 03:26:04.142147 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:04.142116    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68bcb74d46-gzvzp"]
Oct 31 03:26:04.248957 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:04.248922    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hp28\" (UniqueName: \"kubernetes.io/projected/92b23de7-2d11-447b-96ea-acc55c6a26ac-kube-api-access-9hp28\") pod \"authorino-68bcb74d46-gzvzp\" (UID: \"92b23de7-2d11-447b-96ea-acc55c6a26ac\") " pod="kuadrant-system/authorino-68bcb74d46-gzvzp"
Oct 31 03:26:04.350333 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:04.350284    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hp28\" (UniqueName: \"kubernetes.io/projected/92b23de7-2d11-447b-96ea-acc55c6a26ac-kube-api-access-9hp28\") pod \"authorino-68bcb74d46-gzvzp\" (UID: \"92b23de7-2d11-447b-96ea-acc55c6a26ac\") " pod="kuadrant-system/authorino-68bcb74d46-gzvzp"
Oct 31 03:26:04.412366 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:04.412293    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hp28\" (UniqueName: \"kubernetes.io/projected/92b23de7-2d11-447b-96ea-acc55c6a26ac-kube-api-access-9hp28\") pod \"authorino-68bcb74d46-gzvzp\" (UID: \"92b23de7-2d11-447b-96ea-acc55c6a26ac\") " pod="kuadrant-system/authorino-68bcb74d46-gzvzp"
Oct 31 03:26:04.472341 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:04.472302    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68bcb74d46-gzvzp"]
Oct 31 03:26:04.472643 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:04.472624    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68bcb74d46-gzvzp"
Oct 31 03:26:04.501913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:04.499075    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-szbkl"]
Oct 31 03:26:04.506390 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:26:04.506357    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74ca5223_2deb_45ed_af0e_422b1cb677d2.slice/crio-4718baa4a8b0479952e7a401653ed5c9ceb68acc0b87a64ed8fd04aa89b4fcc7 WatchSource:0}: Error finding container 4718baa4a8b0479952e7a401653ed5c9ceb68acc0b87a64ed8fd04aa89b4fcc7: Status 404 returned error can't find the container with id 4718baa4a8b0479952e7a401653ed5c9ceb68acc0b87a64ed8fd04aa89b4fcc7
Oct 31 03:26:04.751647 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:04.751611    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-cbh4k" event={"ID":"da0b818f-0cfd-4c31-a808-588b3138c91b","Type":"ContainerStarted","Data":"9a94fc860809e1432f6107840590bfc81d5aaa0c7c32a502f4ac3bcbe19203bb"}
Oct 31 03:26:04.752639 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:04.752567    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-cbh4k"
Oct 31 03:26:04.755129 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:04.755100    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-szbkl" event={"ID":"74ca5223-2deb-45ed-af0e-422b1cb677d2","Type":"ContainerStarted","Data":"4718baa4a8b0479952e7a401653ed5c9ceb68acc0b87a64ed8fd04aa89b4fcc7"}
Oct 31 03:26:04.871789 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:04.871739    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-cbh4k" podStartSLOduration=1.570224943 podStartE2EDuration="5.871724944s" totalImagesPullingTime="4.301500001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:04.868566891 +0000 UTC m=+390.379549270" watchObservedRunningTime="2026-10-31 03:26:04.871724944 +0000 UTC m=+390.382707320"
Oct 31 03:26:04.935031 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:04.935006    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68bcb74d46-gzvzp"]
Oct 31 03:26:04.959733 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:26:04.959700    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92b23de7_2d11_447b_96ea_acc55c6a26ac.slice/crio-9f6602458ca483f80d21cdea53f197bd990a4df97fcdc6a3f02a8f103a7095c2 WatchSource:0}: Error finding container 9f6602458ca483f80d21cdea53f197bd990a4df97fcdc6a3f02a8f103a7095c2: Status 404 returned error can't find the container with id 9f6602458ca483f80d21cdea53f197bd990a4df97fcdc6a3f02a8f103a7095c2
Oct 31 03:26:05.771645 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:05.771603    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-vvkxg" event={"ID":"c9241326-bbbe-4a12-aabd-67e56ee8e55e","Type":"ContainerStarted","Data":"140bc525639a6c06c05d70960292e8b7e3b7b060332e0044806e9042da2d4ca8"}
Oct 31 03:26:05.772231 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:05.771842    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-vvkxg"
Oct 31 03:26:05.772950 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:05.772924    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68bcb74d46-gzvzp" event={"ID":"92b23de7-2d11-447b-96ea-acc55c6a26ac","Type":"ContainerStarted","Data":"9f6602458ca483f80d21cdea53f197bd990a4df97fcdc6a3f02a8f103a7095c2"}
Oct 31 03:26:06.779606 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:06.779576    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-fh6m6" event={"ID":"73b2caea-8bec-45ab-ad59-0415870d7b05","Type":"ContainerStarted","Data":"b890239e5b6dc040423e495f77f8d793e4243d2d4a9199f343ec4ad939bf148b"}
Oct 31 03:26:06.780088 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:06.779643    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-fh6m6"
Oct 31 03:26:06.781269 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:06.781233    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-szbkl" event={"ID":"74ca5223-2deb-45ed-af0e-422b1cb677d2","Type":"ContainerStarted","Data":"5bd77f579b939be1aefb78d9f166bf45304c51d8855e42390509bed1a03a51ea"}
Oct 31 03:26:06.781397 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:06.781301    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-szbkl" podUID="74ca5223-2deb-45ed-af0e-422b1cb677d2" containerName="authorino" containerID="cri-o://5bd77f579b939be1aefb78d9f166bf45304c51d8855e42390509bed1a03a51ea" gracePeriod=30
Oct 31 03:26:06.897569 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:06.897519    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-fh6m6" podStartSLOduration=2.5817208149999997 podStartE2EDuration="7.897502842s" totalImagesPullingTime="5.315782027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:06.896530392 +0000 UTC m=+392.407512770" watchObservedRunningTime="2026-10-31 03:26:06.897502842 +0000 UTC m=+392.408485219"
Oct 31 03:26:06.898251 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:06.898224    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-vvkxg" podStartSLOduration=3.135488533 podStartE2EDuration="7.898214671s" totalImagesPullingTime="4.762726138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:05.888561591 +0000 UTC m=+391.399543969" watchObservedRunningTime="2026-10-31 03:26:06.898214671 +0000 UTC m=+392.409197071"
Oct 31 03:26:07.016311 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.016239    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-szbkl" podStartSLOduration=1.937636761 podStartE2EDuration="4.016206605s" totalImagesPullingTime="2.078569844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:07.014804504 +0000 UTC m=+392.525786883" watchObservedRunningTime="2026-10-31 03:26:07.016206605 +0000 UTC m=+392.527188983"
Oct 31 03:26:07.043191 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.043156    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-skhfw"]
Oct 31 03:26:07.043440 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.043397    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-skhfw" podUID="366b75db-ad1b-4ab3-b8c2-0b99cd8c0568" containerName="authorino" containerID="cri-o://509b52ab1bb39903ce6edf7f0719f74c0f361ca8d050e01d77c270d239df489a" gracePeriod=30
Oct 31 03:26:07.089079 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.089051    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-szbkl"
Oct 31 03:26:07.183556 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.183526    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ca5223-2deb-45ed-af0e-422b1cb677d2-kube-api-access-j2t4g\" (UniqueName: \"kubernetes.io/projected/74ca5223-2deb-45ed-af0e-422b1cb677d2-kube-api-access-j2t4g\") pod \"74ca5223-2deb-45ed-af0e-422b1cb677d2\" (UID: \"74ca5223-2deb-45ed-af0e-422b1cb677d2\") "
Oct 31 03:26:07.185711 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.185684    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ca5223-2deb-45ed-af0e-422b1cb677d2-kube-api-access-j2t4g" pod "74ca5223-2deb-45ed-af0e-422b1cb677d2" (UID: "74ca5223-2deb-45ed-af0e-422b1cb677d2"). InnerVolumeSpecName "kube-api-access-j2t4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:26:07.285230 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.285199    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2t4g\" (UniqueName: \"kubernetes.io/projected/74ca5223-2deb-45ed-af0e-422b1cb677d2-kube-api-access-j2t4g\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:26:07.368237 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.368201    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-skhfw"
Oct 31 03:26:07.491909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.487119    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/366b75db-ad1b-4ab3-b8c2-0b99cd8c0568-kube-api-access-qcjd7\" (UniqueName: \"kubernetes.io/projected/366b75db-ad1b-4ab3-b8c2-0b99cd8c0568-kube-api-access-qcjd7\") pod \"366b75db-ad1b-4ab3-b8c2-0b99cd8c0568\" (UID: \"366b75db-ad1b-4ab3-b8c2-0b99cd8c0568\") "
Oct 31 03:26:07.491909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.490573    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/366b75db-ad1b-4ab3-b8c2-0b99cd8c0568-kube-api-access-qcjd7" pod "366b75db-ad1b-4ab3-b8c2-0b99cd8c0568" (UID: "366b75db-ad1b-4ab3-b8c2-0b99cd8c0568"). InnerVolumeSpecName "kube-api-access-qcjd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:26:07.593335 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.593265    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcjd7\" (UniqueName: \"kubernetes.io/projected/366b75db-ad1b-4ab3-b8c2-0b99cd8c0568-kube-api-access-qcjd7\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:26:07.790135 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.790047    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-9qtng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:26:07.790135 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.790106    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng" podUID="7c79e465-b30e-482a-bccb-61f5bcb4d910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:26:07.792537 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.792507    4447 generic.go:356] "Generic (PLEG): container finished" podID="74ca5223-2deb-45ed-af0e-422b1cb677d2" containerID="5bd77f579b939be1aefb78d9f166bf45304c51d8855e42390509bed1a03a51ea" exitCode=0
Oct 31 03:26:07.792982 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.792564    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-szbkl"
Oct 31 03:26:07.792982 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.792606    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-szbkl" event={"ID":"74ca5223-2deb-45ed-af0e-422b1cb677d2","Type":"ContainerDied","Data":"5bd77f579b939be1aefb78d9f166bf45304c51d8855e42390509bed1a03a51ea"}
Oct 31 03:26:07.792982 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.792659    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-szbkl" event={"ID":"74ca5223-2deb-45ed-af0e-422b1cb677d2","Type":"ContainerDied","Data":"4718baa4a8b0479952e7a401653ed5c9ceb68acc0b87a64ed8fd04aa89b4fcc7"}
Oct 31 03:26:07.792982 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.792682    4447 scope.go:122] "RemoveContainer" containerID="5bd77f579b939be1aefb78d9f166bf45304c51d8855e42390509bed1a03a51ea"
Oct 31 03:26:07.795655 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.795615    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68bcb74d46-gzvzp" event={"ID":"92b23de7-2d11-447b-96ea-acc55c6a26ac","Type":"ContainerStarted","Data":"0cd7190d6d2268a0b06a8082ba16102d29957246af7dd86c20ac3910335921e1"}
Oct 31 03:26:07.795785 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.795761    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-68bcb74d46-gzvzp" podUID="92b23de7-2d11-447b-96ea-acc55c6a26ac" containerName="authorino" containerID="cri-o://0cd7190d6d2268a0b06a8082ba16102d29957246af7dd86c20ac3910335921e1" gracePeriod=30
Oct 31 03:26:07.797586 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.797560    4447 generic.go:356] "Generic (PLEG): container finished" podID="366b75db-ad1b-4ab3-b8c2-0b99cd8c0568" containerID="509b52ab1bb39903ce6edf7f0719f74c0f361ca8d050e01d77c270d239df489a" exitCode=0
Oct 31 03:26:07.798496 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.798239    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-skhfw"
Oct 31 03:26:07.799981 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.799944    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-skhfw" event={"ID":"366b75db-ad1b-4ab3-b8c2-0b99cd8c0568","Type":"ContainerDied","Data":"509b52ab1bb39903ce6edf7f0719f74c0f361ca8d050e01d77c270d239df489a"}
Oct 31 03:26:07.800077 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.799991    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-skhfw" event={"ID":"366b75db-ad1b-4ab3-b8c2-0b99cd8c0568","Type":"ContainerDied","Data":"9fd9e0a8af4ad9449fd571b896fdcdbe46a4a2e642944a8f4698ee692420931b"}
Oct 31 03:26:07.808025 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.808004    4447 scope.go:122] "RemoveContainer" containerID="5bd77f579b939be1aefb78d9f166bf45304c51d8855e42390509bed1a03a51ea"
Oct 31 03:26:07.808362 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:26:07.808341    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bd77f579b939be1aefb78d9f166bf45304c51d8855e42390509bed1a03a51ea\": container with ID starting with 5bd77f579b939be1aefb78d9f166bf45304c51d8855e42390509bed1a03a51ea not found: ID does not exist" containerID="5bd77f579b939be1aefb78d9f166bf45304c51d8855e42390509bed1a03a51ea"
Oct 31 03:26:07.808422 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.808373    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bd77f579b939be1aefb78d9f166bf45304c51d8855e42390509bed1a03a51ea"} err="failed to get container status \"5bd77f579b939be1aefb78d9f166bf45304c51d8855e42390509bed1a03a51ea\": rpc error: code = NotFound desc = could not find container \"5bd77f579b939be1aefb78d9f166bf45304c51d8855e42390509bed1a03a51ea\": container with ID starting with 5bd77f579b939be1aefb78d9f166bf45304c51d8855e42390509bed1a03a51ea not found: ID does not exist"
Oct 31 03:26:07.808422 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.808391    4447 scope.go:122] "RemoveContainer" containerID="509b52ab1bb39903ce6edf7f0719f74c0f361ca8d050e01d77c270d239df489a"
Oct 31 03:26:07.819579 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.819550    4447 scope.go:122] "RemoveContainer" containerID="509b52ab1bb39903ce6edf7f0719f74c0f361ca8d050e01d77c270d239df489a"
Oct 31 03:26:07.819974 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:26:07.819940    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"509b52ab1bb39903ce6edf7f0719f74c0f361ca8d050e01d77c270d239df489a\": container with ID starting with 509b52ab1bb39903ce6edf7f0719f74c0f361ca8d050e01d77c270d239df489a not found: ID does not exist" containerID="509b52ab1bb39903ce6edf7f0719f74c0f361ca8d050e01d77c270d239df489a"
Oct 31 03:26:07.820063 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.819977    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509b52ab1bb39903ce6edf7f0719f74c0f361ca8d050e01d77c270d239df489a"} err="failed to get container status \"509b52ab1bb39903ce6edf7f0719f74c0f361ca8d050e01d77c270d239df489a\": rpc error: code = NotFound desc = could not find container \"509b52ab1bb39903ce6edf7f0719f74c0f361ca8d050e01d77c270d239df489a\": container with ID starting with 509b52ab1bb39903ce6edf7f0719f74c0f361ca8d050e01d77c270d239df489a not found: ID does not exist"
Oct 31 03:26:07.960346 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.960275    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-skhfw"]
Oct 31 03:26:07.964810 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:07.964774    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-skhfw"]
Oct 31 03:26:08.072637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:08.072608    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-szbkl"]
Oct 31 03:26:08.076335 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:08.076306    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-szbkl"]
Oct 31 03:26:08.120671 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:08.120645    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68bcb74d46-gzvzp"
Oct 31 03:26:08.189512 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:08.189461    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-68bcb74d46-gzvzp" podStartSLOduration=2.379059306 podStartE2EDuration="4.189444156s" totalImagesPullingTime="1.81038485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:08.187131856 +0000 UTC m=+393.698114235" watchObservedRunningTime="2026-10-31 03:26:08.189444156 +0000 UTC m=+393.700426533"
Oct 31 03:26:08.199431 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:08.199402    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92b23de7-2d11-447b-96ea-acc55c6a26ac-kube-api-access-9hp28\" (UniqueName: \"kubernetes.io/projected/92b23de7-2d11-447b-96ea-acc55c6a26ac-kube-api-access-9hp28\") pod \"92b23de7-2d11-447b-96ea-acc55c6a26ac\" (UID: \"92b23de7-2d11-447b-96ea-acc55c6a26ac\") "
Oct 31 03:26:08.201691 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:08.201658    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b23de7-2d11-447b-96ea-acc55c6a26ac-kube-api-access-9hp28" pod "92b23de7-2d11-447b-96ea-acc55c6a26ac" (UID: "92b23de7-2d11-447b-96ea-acc55c6a26ac"). InnerVolumeSpecName "kube-api-access-9hp28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:26:08.300481 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:08.300449    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hp28\" (UniqueName: \"kubernetes.io/projected/92b23de7-2d11-447b-96ea-acc55c6a26ac-kube-api-access-9hp28\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:26:08.610778 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:08.610746    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="366b75db-ad1b-4ab3-b8c2-0b99cd8c0568" path="/var/lib/kubelet/pods/366b75db-ad1b-4ab3-b8c2-0b99cd8c0568/volumes"
Oct 31 03:26:08.611138 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:08.611124    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ca5223-2deb-45ed-af0e-422b1cb677d2" path="/var/lib/kubelet/pods/74ca5223-2deb-45ed-af0e-422b1cb677d2/volumes"
Oct 31 03:26:08.806025 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:08.805994    4447 generic.go:356] "Generic (PLEG): container finished" podID="92b23de7-2d11-447b-96ea-acc55c6a26ac" containerID="0cd7190d6d2268a0b06a8082ba16102d29957246af7dd86c20ac3910335921e1" exitCode=0
Oct 31 03:26:08.806511 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:08.806046    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68bcb74d46-gzvzp"
Oct 31 03:26:08.806511 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:08.806071    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68bcb74d46-gzvzp" event={"ID":"92b23de7-2d11-447b-96ea-acc55c6a26ac","Type":"ContainerDied","Data":"0cd7190d6d2268a0b06a8082ba16102d29957246af7dd86c20ac3910335921e1"}
Oct 31 03:26:08.806511 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:08.806113    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68bcb74d46-gzvzp" event={"ID":"92b23de7-2d11-447b-96ea-acc55c6a26ac","Type":"ContainerDied","Data":"9f6602458ca483f80d21cdea53f197bd990a4df97fcdc6a3f02a8f103a7095c2"}
Oct 31 03:26:08.806511 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:08.806137    4447 scope.go:122] "RemoveContainer" containerID="0cd7190d6d2268a0b06a8082ba16102d29957246af7dd86c20ac3910335921e1"
Oct 31 03:26:08.815950 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:08.815925    4447 scope.go:122] "RemoveContainer" containerID="0cd7190d6d2268a0b06a8082ba16102d29957246af7dd86c20ac3910335921e1"
Oct 31 03:26:08.816210 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:26:08.816190    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cd7190d6d2268a0b06a8082ba16102d29957246af7dd86c20ac3910335921e1\": container with ID starting with 0cd7190d6d2268a0b06a8082ba16102d29957246af7dd86c20ac3910335921e1 not found: ID does not exist" containerID="0cd7190d6d2268a0b06a8082ba16102d29957246af7dd86c20ac3910335921e1"
Oct 31 03:26:08.816282 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:08.816218    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cd7190d6d2268a0b06a8082ba16102d29957246af7dd86c20ac3910335921e1"} err="failed to get container status \"0cd7190d6d2268a0b06a8082ba16102d29957246af7dd86c20ac3910335921e1\": rpc error: code = NotFound desc = could not find container \"0cd7190d6d2268a0b06a8082ba16102d29957246af7dd86c20ac3910335921e1\": container with ID starting with 0cd7190d6d2268a0b06a8082ba16102d29957246af7dd86c20ac3910335921e1 not found: ID does not exist"
Oct 31 03:26:08.978806 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:08.978772    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68bcb74d46-gzvzp"]
Oct 31 03:26:08.980791 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:08.980771    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-68bcb74d46-gzvzp"]
Oct 31 03:26:09.814478 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:09.814447    4447 generic.go:356] "Generic (PLEG): container finished" podID="7c79e465-b30e-482a-bccb-61f5bcb4d910" containerID="c7653d66d44d682f4e02f0195c77262823daf9c3dc1e17068cbe803a7d626110" exitCode=0
Oct 31 03:26:09.814954 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:09.814526    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng" event={"ID":"7c79e465-b30e-482a-bccb-61f5bcb4d910","Type":"ContainerDied","Data":"c7653d66d44d682f4e02f0195c77262823daf9c3dc1e17068cbe803a7d626110"}
Oct 31 03:26:09.815039 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:09.815021    4447 scope.go:122] "RemoveContainer" containerID="c7653d66d44d682f4e02f0195c77262823daf9c3dc1e17068cbe803a7d626110"
Oct 31 03:26:09.816185 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:09.816154    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:26:09.816262 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:09.816228    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:26:09.816314 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:09.816260    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:26:10.607996 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:10.607967    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92b23de7-2d11-447b-96ea-acc55c6a26ac" path="/var/lib/kubelet/pods/92b23de7-2d11-447b-96ea-acc55c6a26ac/volumes"
Oct 31 03:26:10.823061 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:10.823028    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng" event={"ID":"7c79e465-b30e-482a-bccb-61f5bcb4d910","Type":"ContainerStarted","Data":"51f52ddfd8b706a51037a519f91d59a73143ec4b41a2c12a6b8781e015a53f23"}
Oct 31 03:26:11.293910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:11.287284    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:26:11.293910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:11.287416    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-9qtng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Oct 31 03:26:11.293910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:11.287464    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng" podUID="7c79e465-b30e-482a-bccb-61f5bcb4d910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Oct 31 03:26:12.283349 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:12.283308    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-9qtng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Oct 31 03:26:12.284035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:12.284001    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng" podUID="7c79e465-b30e-482a-bccb-61f5bcb4d910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Oct 31 03:26:13.294395 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:13.294359    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-9qtng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Oct 31 03:26:13.295018 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:13.294982    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng" podUID="7c79e465-b30e-482a-bccb-61f5bcb4d910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Oct 31 03:26:13.615609 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:13.615522    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl"]
Oct 31 03:26:13.615819 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:13.615772    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl" podUID="a0e0f40c-fb0c-4941-ba96-13f44169a644" containerName="maas-api" containerID="cri-o://29e5c3d6d2012ffad0a756c4e0b9d8a9c4b1b14e062a5b3bbc330f5ad0fd5976" gracePeriod=30
Oct 31 03:26:13.843842 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:13.843806    4447 generic.go:356] "Generic (PLEG): container finished" podID="a0e0f40c-fb0c-4941-ba96-13f44169a644" containerID="29e5c3d6d2012ffad0a756c4e0b9d8a9c4b1b14e062a5b3bbc330f5ad0fd5976" exitCode=0
Oct 31 03:26:13.844023 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:13.843875    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl" event={"ID":"a0e0f40c-fb0c-4941-ba96-13f44169a644","Type":"ContainerDied","Data":"29e5c3d6d2012ffad0a756c4e0b9d8a9c4b1b14e062a5b3bbc330f5ad0fd5976"}
Oct 31 03:26:13.912151 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:13.912125    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl"
Oct 31 03:26:13.955633 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:13.955596    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0e0f40c-fb0c-4941-ba96-13f44169a644-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0e0f40c-fb0c-4941-ba96-13f44169a644-maas-api-tls\") pod \"a0e0f40c-fb0c-4941-ba96-13f44169a644\" (UID: \"a0e0f40c-fb0c-4941-ba96-13f44169a644\") "
Oct 31 03:26:13.955833 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:13.955698    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0e0f40c-fb0c-4941-ba96-13f44169a644-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0e0f40c-fb0c-4941-ba96-13f44169a644-metrics-certs\") pod \"a0e0f40c-fb0c-4941-ba96-13f44169a644\" (UID: \"a0e0f40c-fb0c-4941-ba96-13f44169a644\") "
Oct 31 03:26:13.955833 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:13.955727    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e0f40c-fb0c-4941-ba96-13f44169a644-kube-api-access-hdfxt\" (UniqueName: \"kubernetes.io/projected/a0e0f40c-fb0c-4941-ba96-13f44169a644-kube-api-access-hdfxt\") pod \"a0e0f40c-fb0c-4941-ba96-13f44169a644\" (UID: \"a0e0f40c-fb0c-4941-ba96-13f44169a644\") "
Oct 31 03:26:13.955833 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:13.955824    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0e0f40c-fb0c-4941-ba96-13f44169a644-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e0f40c-fb0c-4941-ba96-13f44169a644-openshift-service-ca\") pod \"a0e0f40c-fb0c-4941-ba96-13f44169a644\" (UID: \"a0e0f40c-fb0c-4941-ba96-13f44169a644\") "
Oct 31 03:26:13.956233 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:13.956207    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e0f40c-fb0c-4941-ba96-13f44169a644-openshift-service-ca" pod "a0e0f40c-fb0c-4941-ba96-13f44169a644" (UID: "a0e0f40c-fb0c-4941-ba96-13f44169a644"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:26:13.958101 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:13.958077    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0e0f40c-fb0c-4941-ba96-13f44169a644-metrics-certs" pod "a0e0f40c-fb0c-4941-ba96-13f44169a644" (UID: "a0e0f40c-fb0c-4941-ba96-13f44169a644"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:26:13.958179 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:13.958114    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0e0f40c-fb0c-4941-ba96-13f44169a644-maas-api-tls" pod "a0e0f40c-fb0c-4941-ba96-13f44169a644" (UID: "a0e0f40c-fb0c-4941-ba96-13f44169a644"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:26:13.958179 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:13.958144    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e0f40c-fb0c-4941-ba96-13f44169a644-kube-api-access-hdfxt" pod "a0e0f40c-fb0c-4941-ba96-13f44169a644" (UID: "a0e0f40c-fb0c-4941-ba96-13f44169a644"). InnerVolumeSpecName "kube-api-access-hdfxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:26:14.057625 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:14.057589    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0e0f40c-fb0c-4941-ba96-13f44169a644-metrics-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:26:14.057625 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:14.057619    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdfxt\" (UniqueName: \"kubernetes.io/projected/a0e0f40c-fb0c-4941-ba96-13f44169a644-kube-api-access-hdfxt\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:26:14.057625 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:14.057629    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e0f40c-fb0c-4941-ba96-13f44169a644-openshift-service-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:26:14.057915 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:14.057639    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0e0f40c-fb0c-4941-ba96-13f44169a644-maas-api-tls\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:26:14.289383 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:14.289340    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-9qtng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Oct 31 03:26:14.289698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:14.289645    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng" podUID="7c79e465-b30e-482a-bccb-61f5bcb4d910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Oct 31 03:26:14.850502 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:14.850467    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl" event={"ID":"a0e0f40c-fb0c-4941-ba96-13f44169a644","Type":"ContainerDied","Data":"38a36f6b521681a870005ead7f189dbd12e70a3fad99dc19774fbd64faec351c"}
Oct 31 03:26:14.850502 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:14.850504    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl"
Oct 31 03:26:14.850969 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:14.850540    4447 scope.go:122] "RemoveContainer" containerID="29e5c3d6d2012ffad0a756c4e0b9d8a9c4b1b14e062a5b3bbc330f5ad0fd5976"
Oct 31 03:26:15.016664 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:15.016625    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl"]
Oct 31 03:26:15.066443 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:15.066404    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fhkjl"]
Oct 31 03:26:15.295967 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:15.288384    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-9qtng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Oct 31 03:26:15.295967 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:15.288460    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng" podUID="7c79e465-b30e-482a-bccb-61f5bcb4d910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Oct 31 03:26:16.286964 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:16.286864    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:26:16.287484 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:16.287464    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:26:16.288260 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:16.288239    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:26:16.608236 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:16.608159    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0e0f40c-fb0c-4941-ba96-13f44169a644" path="/var/lib/kubelet/pods/a0e0f40c-fb0c-4941-ba96-13f44169a644/volumes"
Oct 31 03:26:16.786379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:16.786350    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-vvkxg"
Oct 31 03:26:17.800785 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:17.800754    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-fh6m6"
Oct 31 03:26:19.002553 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:26:19.002517    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:26:36.786641 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:26:36.786607    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-cbh4k"
Oct 31 03:26:49.002721 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:26:49.002680    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:27:19.001094 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:27:19.001055    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:27:49.001797 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:27:49.001715    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:28:05.932313 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:05.932270    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"]
Oct 31 03:28:05.933008 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:05.932989    4447 cpu_manager.go:414] "containerMap: removing container" podUID="366b75db-ad1b-4ab3-b8c2-0b99cd8c0568" containerName="authorino"
Oct 31 03:28:05.933105 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:05.933010    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="366b75db-ad1b-4ab3-b8c2-0b99cd8c0568" containerName="authorino"
Oct 31 03:28:05.933105 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:05.933040    4447 cpu_manager.go:414] "containerMap: removing container" podUID="a0e0f40c-fb0c-4941-ba96-13f44169a644" containerName="maas-api"
Oct 31 03:28:05.933105 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:05.933048    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e0f40c-fb0c-4941-ba96-13f44169a644" containerName="maas-api"
Oct 31 03:28:05.933105 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:05.933063    4447 cpu_manager.go:414] "containerMap: removing container" podUID="74ca5223-2deb-45ed-af0e-422b1cb677d2" containerName="authorino"
Oct 31 03:28:05.933105 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:05.933070    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ca5223-2deb-45ed-af0e-422b1cb677d2" containerName="authorino"
Oct 31 03:28:05.933105 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:05.933097    4447 cpu_manager.go:414] "containerMap: removing container" podUID="92b23de7-2d11-447b-96ea-acc55c6a26ac" containerName="authorino"
Oct 31 03:28:05.933105 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:05.933104    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92b23de7-2d11-447b-96ea-acc55c6a26ac" containerName="authorino"
Oct 31 03:28:05.933431 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:05.933205    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="92b23de7-2d11-447b-96ea-acc55c6a26ac" containerName="authorino"
Oct 31 03:28:05.933431 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:05.933221    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="366b75db-ad1b-4ab3-b8c2-0b99cd8c0568" containerName="authorino"
Oct 31 03:28:05.933431 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:05.933232    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e0f40c-fb0c-4941-ba96-13f44169a644" containerName="maas-api"
Oct 31 03:28:05.933431 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:05.933242    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ca5223-2deb-45ed-af0e-422b1cb677d2" containerName="authorino"
Oct 31 03:28:05.935281 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:05.935260    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"
Oct 31 03:28:05.987804 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:05.987774    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 03:28:06.021521 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:06.021485    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98f65062-e7db-4f54-9d76-6addebbebae6-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nbrhn\" (UID: \"98f65062-e7db-4f54-9d76-6addebbebae6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"
Oct 31 03:28:06.021682 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:06.021548    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbm6m\" (UniqueName: \"kubernetes.io/projected/98f65062-e7db-4f54-9d76-6addebbebae6-kube-api-access-dbm6m\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nbrhn\" (UID: \"98f65062-e7db-4f54-9d76-6addebbebae6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"
Oct 31 03:28:06.021682 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:06.021646    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98f65062-e7db-4f54-9d76-6addebbebae6-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nbrhn\" (UID: \"98f65062-e7db-4f54-9d76-6addebbebae6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"
Oct 31 03:28:06.021682 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:06.021668    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98f65062-e7db-4f54-9d76-6addebbebae6-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nbrhn\" (UID: \"98f65062-e7db-4f54-9d76-6addebbebae6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"
Oct 31 03:28:06.021816 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:06.021694    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98f65062-e7db-4f54-9d76-6addebbebae6-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nbrhn\" (UID: \"98f65062-e7db-4f54-9d76-6addebbebae6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"
Oct 31 03:28:06.021816 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:06.021767    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98f65062-e7db-4f54-9d76-6addebbebae6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nbrhn\" (UID: \"98f65062-e7db-4f54-9d76-6addebbebae6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"
Oct 31 03:28:06.037422 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:06.037397    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 31 03:28:06.037422 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:06.037417    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 03:28:06.037579 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:06.037438    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-j8tx4\""
Oct 31 03:28:06.039936 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:06.039917    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"]
Oct 31 03:28:06.122650 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:06.122622    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98f65062-e7db-4f54-9d76-6addebbebae6-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nbrhn\" (UID: \"98f65062-e7db-4f54-9d76-6addebbebae6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"
Oct 31 03:28:06.122650 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:06.122651    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98f65062-e7db-4f54-9d76-6addebbebae6-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nbrhn\" (UID: \"98f65062-e7db-4f54-9d76-6addebbebae6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"
Oct 31 03:28:06.122927 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:06.122677    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98f65062-e7db-4f54-9d76-6addebbebae6-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nbrhn\" (UID: \"98f65062-e7db-4f54-9d76-6addebbebae6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"
Oct 31 03:28:06.122927 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:06.122796    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98f65062-e7db-4f54-9d76-6addebbebae6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nbrhn\" (UID: \"98f65062-e7db-4f54-9d76-6addebbebae6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"
Oct 31 03:28:06.122927 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:06.122869    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98f65062-e7db-4f54-9d76-6addebbebae6-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nbrhn\" (UID: \"98f65062-e7db-4f54-9d76-6addebbebae6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"
Oct 31 03:28:06.123089 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:06.122965    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbm6m\" (UniqueName: \"kubernetes.io/projected/98f65062-e7db-4f54-9d76-6addebbebae6-kube-api-access-dbm6m\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nbrhn\" (UID: \"98f65062-e7db-4f54-9d76-6addebbebae6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"
Oct 31 03:28:06.123176 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:06.123154    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98f65062-e7db-4f54-9d76-6addebbebae6-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nbrhn\" (UID: \"98f65062-e7db-4f54-9d76-6addebbebae6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"
Oct 31 03:28:06.123236 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:06.123200    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98f65062-e7db-4f54-9d76-6addebbebae6-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nbrhn\" (UID: \"98f65062-e7db-4f54-9d76-6addebbebae6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"
Oct 31 03:28:06.123278 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:06.123247    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98f65062-e7db-4f54-9d76-6addebbebae6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nbrhn\" (UID: \"98f65062-e7db-4f54-9d76-6addebbebae6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"
Oct 31 03:28:06.125163 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:06.125140    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98f65062-e7db-4f54-9d76-6addebbebae6-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nbrhn\" (UID: \"98f65062-e7db-4f54-9d76-6addebbebae6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"
Oct 31 03:28:06.125352 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:06.125336    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98f65062-e7db-4f54-9d76-6addebbebae6-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nbrhn\" (UID: \"98f65062-e7db-4f54-9d76-6addebbebae6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"
Oct 31 03:28:06.184552 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:06.184490    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbm6m\" (UniqueName: \"kubernetes.io/projected/98f65062-e7db-4f54-9d76-6addebbebae6-kube-api-access-dbm6m\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-nbrhn\" (UID: \"98f65062-e7db-4f54-9d76-6addebbebae6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"
Oct 31 03:28:06.246753 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:06.246721    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"
Oct 31 03:28:06.622418 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:06.622397    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"]
Oct 31 03:28:06.623747 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:28:06.623717    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f65062_e7db_4f54_9d76_6addebbebae6.slice/crio-614a8750774b1674a8a64b29d28baadf699bc65e8f2671536fc6c4e84679fbdb WatchSource:0}: Error finding container 614a8750774b1674a8a64b29d28baadf699bc65e8f2671536fc6c4e84679fbdb: Status 404 returned error can't find the container with id 614a8750774b1674a8a64b29d28baadf699bc65e8f2671536fc6c4e84679fbdb
Oct 31 03:28:07.433877 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:07.433838    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn" event={"ID":"98f65062-e7db-4f54-9d76-6addebbebae6","Type":"ContainerStarted","Data":"614a8750774b1674a8a64b29d28baadf699bc65e8f2671536fc6c4e84679fbdb"}
Oct 31 03:28:09.446744 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:09.446708    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn" event={"ID":"98f65062-e7db-4f54-9d76-6addebbebae6","Type":"ContainerStarted","Data":"602fa1fea2fd93d377490a5f1ba704da91a9b64bcbd7d512e4d2cd909d7f0677"}
Oct 31 03:28:09.598585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:09.598528    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn" podStartSLOduration=2.518180846 podStartE2EDuration="4.598515348s" totalImagesPullingTime="2.080334502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:09.59752948 +0000 UTC m=+515.108511867" watchObservedRunningTime="2026-10-31 03:28:09.598515348 +0000 UTC m=+515.109497725"
Oct 31 03:28:10.247431 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:10.247395    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"
Oct 31 03:28:13.211901 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:13.211849    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"]
Oct 31 03:28:13.215492 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:13.215476    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"
Oct 31 03:28:13.289346 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:13.289310    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3767bbe-f08f-47b5-84eb-f01ebf697a41-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xl2pv\" (UID: \"a3767bbe-f08f-47b5-84eb-f01ebf697a41\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"
Oct 31 03:28:13.289346 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:13.289344    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3767bbe-f08f-47b5-84eb-f01ebf697a41-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xl2pv\" (UID: \"a3767bbe-f08f-47b5-84eb-f01ebf697a41\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"
Oct 31 03:28:13.289541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:13.289366    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3767bbe-f08f-47b5-84eb-f01ebf697a41-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xl2pv\" (UID: \"a3767bbe-f08f-47b5-84eb-f01ebf697a41\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"
Oct 31 03:28:13.289541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:13.289389    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3767bbe-f08f-47b5-84eb-f01ebf697a41-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xl2pv\" (UID: \"a3767bbe-f08f-47b5-84eb-f01ebf697a41\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"
Oct 31 03:28:13.289609 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:13.289537    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3767bbe-f08f-47b5-84eb-f01ebf697a41-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xl2pv\" (UID: \"a3767bbe-f08f-47b5-84eb-f01ebf697a41\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"
Oct 31 03:28:13.289609 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:13.289563    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlfjw\" (UniqueName: \"kubernetes.io/projected/a3767bbe-f08f-47b5-84eb-f01ebf697a41-kube-api-access-nlfjw\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xl2pv\" (UID: \"a3767bbe-f08f-47b5-84eb-f01ebf697a41\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"
Oct 31 03:28:13.317612 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:13.317581    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 31 03:28:13.320133 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:13.320112    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"]
Oct 31 03:28:13.391071 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:13.391037    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3767bbe-f08f-47b5-84eb-f01ebf697a41-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xl2pv\" (UID: \"a3767bbe-f08f-47b5-84eb-f01ebf697a41\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"
Oct 31 03:28:13.391071 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:13.391073    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlfjw\" (UniqueName: \"kubernetes.io/projected/a3767bbe-f08f-47b5-84eb-f01ebf697a41-kube-api-access-nlfjw\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xl2pv\" (UID: \"a3767bbe-f08f-47b5-84eb-f01ebf697a41\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"
Oct 31 03:28:13.391329 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:13.391138    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3767bbe-f08f-47b5-84eb-f01ebf697a41-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xl2pv\" (UID: \"a3767bbe-f08f-47b5-84eb-f01ebf697a41\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"
Oct 31 03:28:13.391329 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:13.391165    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3767bbe-f08f-47b5-84eb-f01ebf697a41-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xl2pv\" (UID: \"a3767bbe-f08f-47b5-84eb-f01ebf697a41\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"
Oct 31 03:28:13.391329 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:13.391197    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3767bbe-f08f-47b5-84eb-f01ebf697a41-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xl2pv\" (UID: \"a3767bbe-f08f-47b5-84eb-f01ebf697a41\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"
Oct 31 03:28:13.391329 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:13.391229    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3767bbe-f08f-47b5-84eb-f01ebf697a41-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xl2pv\" (UID: \"a3767bbe-f08f-47b5-84eb-f01ebf697a41\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"
Oct 31 03:28:13.391578 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:13.391552    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3767bbe-f08f-47b5-84eb-f01ebf697a41-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xl2pv\" (UID: \"a3767bbe-f08f-47b5-84eb-f01ebf697a41\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"
Oct 31 03:28:13.391633 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:13.391594    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3767bbe-f08f-47b5-84eb-f01ebf697a41-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xl2pv\" (UID: \"a3767bbe-f08f-47b5-84eb-f01ebf697a41\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"
Oct 31 03:28:13.391684 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:13.391653    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3767bbe-f08f-47b5-84eb-f01ebf697a41-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xl2pv\" (UID: \"a3767bbe-f08f-47b5-84eb-f01ebf697a41\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"
Oct 31 03:28:13.393401 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:13.393384    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3767bbe-f08f-47b5-84eb-f01ebf697a41-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xl2pv\" (UID: \"a3767bbe-f08f-47b5-84eb-f01ebf697a41\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"
Oct 31 03:28:13.393700 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:13.393685    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3767bbe-f08f-47b5-84eb-f01ebf697a41-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xl2pv\" (UID: \"a3767bbe-f08f-47b5-84eb-f01ebf697a41\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"
Oct 31 03:28:13.452867 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:13.452839    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlfjw\" (UniqueName: \"kubernetes.io/projected/a3767bbe-f08f-47b5-84eb-f01ebf697a41-kube-api-access-nlfjw\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xl2pv\" (UID: \"a3767bbe-f08f-47b5-84eb-f01ebf697a41\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"
Oct 31 03:28:13.528644 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:13.528556    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"
Oct 31 03:28:13.906383 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:13.906354    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"]
Oct 31 03:28:13.908027 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:28:13.908001    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3767bbe_f08f_47b5_84eb_f01ebf697a41.slice/crio-0e3e9fce8fc774eaabb589fc05699bfc6a50b1ab243f03b1ef966b975f0bd31a WatchSource:0}: Error finding container 0e3e9fce8fc774eaabb589fc05699bfc6a50b1ab243f03b1ef966b975f0bd31a: Status 404 returned error can't find the container with id 0e3e9fce8fc774eaabb589fc05699bfc6a50b1ab243f03b1ef966b975f0bd31a
Oct 31 03:28:14.473757 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:14.473724    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv" event={"ID":"a3767bbe-f08f-47b5-84eb-f01ebf697a41","Type":"ContainerStarted","Data":"0e3e9fce8fc774eaabb589fc05699bfc6a50b1ab243f03b1ef966b975f0bd31a"}
Oct 31 03:28:15.479898 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:15.479859    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv" event={"ID":"a3767bbe-f08f-47b5-84eb-f01ebf697a41","Type":"ContainerStarted","Data":"0161cb04a81f0547330dbbbd31343712a9b9dcb6ca7792d6bc0af7e08b9ae655"}
Oct 31 03:28:15.529458 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:15.529422    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"
Oct 31 03:28:15.600431 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:15.600363    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv" podStartSLOduration=1.9953325720000001 podStartE2EDuration="2.600344685s" totalImagesPullingTime="605.012113ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:15.597208595 +0000 UTC m=+521.108190996" watchObservedRunningTime="2026-10-31 03:28:15.600344685 +0000 UTC m=+521.111327065"
Oct 31 03:28:16.247066 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.247031    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"
Oct 31 03:28:16.259364 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.259336    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"
Oct 31 03:28:16.366468 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.366435    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"]
Oct 31 03:28:16.370207 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.370191    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"
Oct 31 03:28:16.421446 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.421411    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be4ef64f-41b3-4ec7-9408-bbbb03f85575-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dsdwd\" (UID: \"be4ef64f-41b3-4ec7-9408-bbbb03f85575\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"
Oct 31 03:28:16.421616 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.421533    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be4ef64f-41b3-4ec7-9408-bbbb03f85575-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dsdwd\" (UID: \"be4ef64f-41b3-4ec7-9408-bbbb03f85575\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"
Oct 31 03:28:16.421679 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.421619    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be4ef64f-41b3-4ec7-9408-bbbb03f85575-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dsdwd\" (UID: \"be4ef64f-41b3-4ec7-9408-bbbb03f85575\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"
Oct 31 03:28:16.421679 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.421639    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be4ef64f-41b3-4ec7-9408-bbbb03f85575-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dsdwd\" (UID: \"be4ef64f-41b3-4ec7-9408-bbbb03f85575\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"
Oct 31 03:28:16.421774 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.421762    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be4ef64f-41b3-4ec7-9408-bbbb03f85575-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dsdwd\" (UID: \"be4ef64f-41b3-4ec7-9408-bbbb03f85575\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"
Oct 31 03:28:16.421823 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.421791    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxm4b\" (UniqueName: \"kubernetes.io/projected/be4ef64f-41b3-4ec7-9408-bbbb03f85575-kube-api-access-sxm4b\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dsdwd\" (UID: \"be4ef64f-41b3-4ec7-9408-bbbb03f85575\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"
Oct 31 03:28:16.426611 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.426372    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 31 03:28:16.431028 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.431003    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"]
Oct 31 03:28:16.497576 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.497494    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn"
Oct 31 03:28:16.522833 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.522796    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be4ef64f-41b3-4ec7-9408-bbbb03f85575-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dsdwd\" (UID: \"be4ef64f-41b3-4ec7-9408-bbbb03f85575\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"
Oct 31 03:28:16.523035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.522975    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be4ef64f-41b3-4ec7-9408-bbbb03f85575-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dsdwd\" (UID: \"be4ef64f-41b3-4ec7-9408-bbbb03f85575\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"
Oct 31 03:28:16.523035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.523018    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be4ef64f-41b3-4ec7-9408-bbbb03f85575-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dsdwd\" (UID: \"be4ef64f-41b3-4ec7-9408-bbbb03f85575\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"
Oct 31 03:28:16.523149 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.523094    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be4ef64f-41b3-4ec7-9408-bbbb03f85575-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dsdwd\" (UID: \"be4ef64f-41b3-4ec7-9408-bbbb03f85575\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"
Oct 31 03:28:16.523149 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.523124    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxm4b\" (UniqueName: \"kubernetes.io/projected/be4ef64f-41b3-4ec7-9408-bbbb03f85575-kube-api-access-sxm4b\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dsdwd\" (UID: \"be4ef64f-41b3-4ec7-9408-bbbb03f85575\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"
Oct 31 03:28:16.523256 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.523214    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be4ef64f-41b3-4ec7-9408-bbbb03f85575-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dsdwd\" (UID: \"be4ef64f-41b3-4ec7-9408-bbbb03f85575\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"
Oct 31 03:28:16.523361 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.523343    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be4ef64f-41b3-4ec7-9408-bbbb03f85575-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dsdwd\" (UID: \"be4ef64f-41b3-4ec7-9408-bbbb03f85575\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"
Oct 31 03:28:16.523602 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.523582    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be4ef64f-41b3-4ec7-9408-bbbb03f85575-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dsdwd\" (UID: \"be4ef64f-41b3-4ec7-9408-bbbb03f85575\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"
Oct 31 03:28:16.524263 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.524241    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be4ef64f-41b3-4ec7-9408-bbbb03f85575-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dsdwd\" (UID: \"be4ef64f-41b3-4ec7-9408-bbbb03f85575\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"
Oct 31 03:28:16.525767 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.525739    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be4ef64f-41b3-4ec7-9408-bbbb03f85575-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dsdwd\" (UID: \"be4ef64f-41b3-4ec7-9408-bbbb03f85575\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"
Oct 31 03:28:16.525916 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.525790    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be4ef64f-41b3-4ec7-9408-bbbb03f85575-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dsdwd\" (UID: \"be4ef64f-41b3-4ec7-9408-bbbb03f85575\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"
Oct 31 03:28:16.584788 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.584761    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxm4b\" (UniqueName: \"kubernetes.io/projected/be4ef64f-41b3-4ec7-9408-bbbb03f85575-kube-api-access-sxm4b\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dsdwd\" (UID: \"be4ef64f-41b3-4ec7-9408-bbbb03f85575\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"
Oct 31 03:28:16.683241 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:16.683207    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"
Oct 31 03:28:17.063617 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:17.063593    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"]
Oct 31 03:28:17.065049 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:28:17.065026    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4ef64f_41b3_4ec7_9408_bbbb03f85575.slice/crio-278380d630e8f229bdd6a2d03032307f77e2b62c30a3bd7844dbc6377094ec2d WatchSource:0}: Error finding container 278380d630e8f229bdd6a2d03032307f77e2b62c30a3bd7844dbc6377094ec2d: Status 404 returned error can't find the container with id 278380d630e8f229bdd6a2d03032307f77e2b62c30a3bd7844dbc6377094ec2d
Oct 31 03:28:17.491410 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:17.491375    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd" event={"ID":"be4ef64f-41b3-4ec7-9408-bbbb03f85575","Type":"ContainerStarted","Data":"278380d630e8f229bdd6a2d03032307f77e2b62c30a3bd7844dbc6377094ec2d"}
Oct 31 03:28:18.498057 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:18.498027    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd" event={"ID":"be4ef64f-41b3-4ec7-9408-bbbb03f85575","Type":"ContainerStarted","Data":"7b772a34bde1bd288a95955fda3c975751fb603abff83e0ef44166b90c97ba07"}
Oct 31 03:28:18.616799 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:18.616750    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd" podStartSLOduration=2.041911215 podStartE2EDuration="2.616735279s" totalImagesPullingTime="574.824064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:18.616294851 +0000 UTC m=+524.127277227" watchObservedRunningTime="2026-10-31 03:28:18.616735279 +0000 UTC m=+524.127717655"
Oct 31 03:28:18.683943 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:18.683912    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"
Oct 31 03:28:19.001442 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:28:19.001401    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:28:19.667037 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:19.667000    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"]
Oct 31 03:28:19.670832 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:19.670811    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"
Oct 31 03:28:19.741392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:19.741357    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 31 03:28:19.743781 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:19.743754    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"]
Oct 31 03:28:19.758839 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:19.758814    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7523be11-f92d-47cc-9309-998e5e4caa6b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r\" (UID: \"7523be11-f92d-47cc-9309-998e5e4caa6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"
Oct 31 03:28:19.758973 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:19.758870    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7523be11-f92d-47cc-9309-998e5e4caa6b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r\" (UID: \"7523be11-f92d-47cc-9309-998e5e4caa6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"
Oct 31 03:28:19.759019 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:19.759004    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7523be11-f92d-47cc-9309-998e5e4caa6b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r\" (UID: \"7523be11-f92d-47cc-9309-998e5e4caa6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"
Oct 31 03:28:19.759071 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:19.759053    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7523be11-f92d-47cc-9309-998e5e4caa6b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r\" (UID: \"7523be11-f92d-47cc-9309-998e5e4caa6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"
Oct 31 03:28:19.759149 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:19.759133    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgd8t\" (UniqueName: \"kubernetes.io/projected/7523be11-f92d-47cc-9309-998e5e4caa6b-kube-api-access-vgd8t\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r\" (UID: \"7523be11-f92d-47cc-9309-998e5e4caa6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"
Oct 31 03:28:19.759183 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:19.759161    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7523be11-f92d-47cc-9309-998e5e4caa6b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r\" (UID: \"7523be11-f92d-47cc-9309-998e5e4caa6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"
Oct 31 03:28:19.859874 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:19.859840    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7523be11-f92d-47cc-9309-998e5e4caa6b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r\" (UID: \"7523be11-f92d-47cc-9309-998e5e4caa6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"
Oct 31 03:28:19.860078 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:19.859919    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7523be11-f92d-47cc-9309-998e5e4caa6b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r\" (UID: \"7523be11-f92d-47cc-9309-998e5e4caa6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"
Oct 31 03:28:19.860078 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:19.859971    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgd8t\" (UniqueName: \"kubernetes.io/projected/7523be11-f92d-47cc-9309-998e5e4caa6b-kube-api-access-vgd8t\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r\" (UID: \"7523be11-f92d-47cc-9309-998e5e4caa6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"
Oct 31 03:28:19.860078 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:19.860002    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7523be11-f92d-47cc-9309-998e5e4caa6b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r\" (UID: \"7523be11-f92d-47cc-9309-998e5e4caa6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"
Oct 31 03:28:19.860078 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:19.860052    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7523be11-f92d-47cc-9309-998e5e4caa6b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r\" (UID: \"7523be11-f92d-47cc-9309-998e5e4caa6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"
Oct 31 03:28:19.860309 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:19.860107    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7523be11-f92d-47cc-9309-998e5e4caa6b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r\" (UID: \"7523be11-f92d-47cc-9309-998e5e4caa6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"
Oct 31 03:28:19.860309 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:19.860239    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7523be11-f92d-47cc-9309-998e5e4caa6b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r\" (UID: \"7523be11-f92d-47cc-9309-998e5e4caa6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"
Oct 31 03:28:19.860444 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:19.860424    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7523be11-f92d-47cc-9309-998e5e4caa6b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r\" (UID: \"7523be11-f92d-47cc-9309-998e5e4caa6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"
Oct 31 03:28:19.860621 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:19.860599    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7523be11-f92d-47cc-9309-998e5e4caa6b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r\" (UID: \"7523be11-f92d-47cc-9309-998e5e4caa6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"
Oct 31 03:28:19.862487 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:19.862466    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7523be11-f92d-47cc-9309-998e5e4caa6b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r\" (UID: \"7523be11-f92d-47cc-9309-998e5e4caa6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"
Oct 31 03:28:19.862671 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:19.862657    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7523be11-f92d-47cc-9309-998e5e4caa6b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r\" (UID: \"7523be11-f92d-47cc-9309-998e5e4caa6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"
Oct 31 03:28:19.922393 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:19.922321    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgd8t\" (UniqueName: \"kubernetes.io/projected/7523be11-f92d-47cc-9309-998e5e4caa6b-kube-api-access-vgd8t\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r\" (UID: \"7523be11-f92d-47cc-9309-998e5e4caa6b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"
Oct 31 03:28:19.984106 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:19.984069    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"
Oct 31 03:28:20.361639 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:20.361615    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"]
Oct 31 03:28:20.362874 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:28:20.362840    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7523be11_f92d_47cc_9309_998e5e4caa6b.slice/crio-42bb80e6ac663a0ead0afa5951b10aa66a05baa1c64e62fafc81b1a1df87d8a6 WatchSource:0}: Error finding container 42bb80e6ac663a0ead0afa5951b10aa66a05baa1c64e62fafc81b1a1df87d8a6: Status 404 returned error can't find the container with id 42bb80e6ac663a0ead0afa5951b10aa66a05baa1c64e62fafc81b1a1df87d8a6
Oct 31 03:28:20.509600 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:20.509571    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r" event={"ID":"7523be11-f92d-47cc-9309-998e5e4caa6b","Type":"ContainerStarted","Data":"42bb80e6ac663a0ead0afa5951b10aa66a05baa1c64e62fafc81b1a1df87d8a6"}
Oct 31 03:28:21.516812 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:21.516772    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r" event={"ID":"7523be11-f92d-47cc-9309-998e5e4caa6b","Type":"ContainerStarted","Data":"dfa5dcb7f39080bf5b7a32d18704dce4dbaea4812ff90d6fac1ba0d55c9b06ff"}
Oct 31 03:28:21.635496 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:21.635442    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r" podStartSLOduration=2.053151309 podStartE2EDuration="2.635421615s" totalImagesPullingTime="582.270306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:21.635165612 +0000 UTC m=+527.146147991" watchObservedRunningTime="2026-10-31 03:28:21.635421615 +0000 UTC m=+527.146403991"
Oct 31 03:28:21.984946 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:21.984910    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"
Oct 31 03:28:23.432359 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.432321    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-s6nfq"]
Oct 31 03:28:23.437111 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.437084    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq"
Oct 31 03:28:23.491788 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.491755    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 31 03:28:23.493476 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.493449    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rns2c\" (UniqueName: \"kubernetes.io/projected/41c075e6-b57b-4e72-ad58-59450e64c8a3-kube-api-access-rns2c\") pod \"echo-simulator-kserve-64b6f8f96c-s6nfq\" (UID: \"41c075e6-b57b-4e72-ad58-59450e64c8a3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq"
Oct 31 03:28:23.493874 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.493853    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41c075e6-b57b-4e72-ad58-59450e64c8a3-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-s6nfq\" (UID: \"41c075e6-b57b-4e72-ad58-59450e64c8a3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq"
Oct 31 03:28:23.494110 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.494091    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41c075e6-b57b-4e72-ad58-59450e64c8a3-home\") pod \"echo-simulator-kserve-64b6f8f96c-s6nfq\" (UID: \"41c075e6-b57b-4e72-ad58-59450e64c8a3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq"
Oct 31 03:28:23.494286 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.494268    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41c075e6-b57b-4e72-ad58-59450e64c8a3-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-s6nfq\" (UID: \"41c075e6-b57b-4e72-ad58-59450e64c8a3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq"
Oct 31 03:28:23.494488 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.494471    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41c075e6-b57b-4e72-ad58-59450e64c8a3-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-s6nfq\" (UID: \"41c075e6-b57b-4e72-ad58-59450e64c8a3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq"
Oct 31 03:28:23.494636 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.494620    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41c075e6-b57b-4e72-ad58-59450e64c8a3-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-s6nfq\" (UID: \"41c075e6-b57b-4e72-ad58-59450e64c8a3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq"
Oct 31 03:28:23.529576 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.529530    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"
Oct 31 03:28:23.543544 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.543513    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-s6nfq"]
Oct 31 03:28:23.546654 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.546633    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"
Oct 31 03:28:23.595830 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.595791    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41c075e6-b57b-4e72-ad58-59450e64c8a3-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-s6nfq\" (UID: \"41c075e6-b57b-4e72-ad58-59450e64c8a3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq"
Oct 31 03:28:23.596062 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.595839    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41c075e6-b57b-4e72-ad58-59450e64c8a3-home\") pod \"echo-simulator-kserve-64b6f8f96c-s6nfq\" (UID: \"41c075e6-b57b-4e72-ad58-59450e64c8a3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq"
Oct 31 03:28:23.596062 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.595860    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41c075e6-b57b-4e72-ad58-59450e64c8a3-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-s6nfq\" (UID: \"41c075e6-b57b-4e72-ad58-59450e64c8a3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq"
Oct 31 03:28:23.596062 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.595971    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41c075e6-b57b-4e72-ad58-59450e64c8a3-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-s6nfq\" (UID: \"41c075e6-b57b-4e72-ad58-59450e64c8a3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq"
Oct 31 03:28:23.596062 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.596005    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41c075e6-b57b-4e72-ad58-59450e64c8a3-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-s6nfq\" (UID: \"41c075e6-b57b-4e72-ad58-59450e64c8a3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq"
Oct 31 03:28:23.596062 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.596056    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rns2c\" (UniqueName: \"kubernetes.io/projected/41c075e6-b57b-4e72-ad58-59450e64c8a3-kube-api-access-rns2c\") pod \"echo-simulator-kserve-64b6f8f96c-s6nfq\" (UID: \"41c075e6-b57b-4e72-ad58-59450e64c8a3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq"
Oct 31 03:28:23.596359 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.596333    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41c075e6-b57b-4e72-ad58-59450e64c8a3-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-s6nfq\" (UID: \"41c075e6-b57b-4e72-ad58-59450e64c8a3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq"
Oct 31 03:28:23.597013 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.596988    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41c075e6-b57b-4e72-ad58-59450e64c8a3-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-s6nfq\" (UID: \"41c075e6-b57b-4e72-ad58-59450e64c8a3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq"
Oct 31 03:28:23.597137 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.597117    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41c075e6-b57b-4e72-ad58-59450e64c8a3-home\") pod \"echo-simulator-kserve-64b6f8f96c-s6nfq\" (UID: \"41c075e6-b57b-4e72-ad58-59450e64c8a3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq"
Oct 31 03:28:23.598758 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.598735    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41c075e6-b57b-4e72-ad58-59450e64c8a3-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-s6nfq\" (UID: \"41c075e6-b57b-4e72-ad58-59450e64c8a3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq"
Oct 31 03:28:23.599375 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.599355    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41c075e6-b57b-4e72-ad58-59450e64c8a3-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-s6nfq\" (UID: \"41c075e6-b57b-4e72-ad58-59450e64c8a3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq"
Oct 31 03:28:23.658134 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.658105    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rns2c\" (UniqueName: \"kubernetes.io/projected/41c075e6-b57b-4e72-ad58-59450e64c8a3-kube-api-access-rns2c\") pod \"echo-simulator-kserve-64b6f8f96c-s6nfq\" (UID: \"41c075e6-b57b-4e72-ad58-59450e64c8a3\") " pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq"
Oct 31 03:28:23.752316 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:23.752285    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq"
Oct 31 03:28:24.140621 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:24.140596    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-s6nfq"]
Oct 31 03:28:24.141975 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:28:24.141951    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c075e6_b57b_4e72_ad58_59450e64c8a3.slice/crio-ec7edec9f853b95b0f1d7782599f34f80dfe34df91870b7f43711aca6424eefd WatchSource:0}: Error finding container ec7edec9f853b95b0f1d7782599f34f80dfe34df91870b7f43711aca6424eefd: Status 404 returned error can't find the container with id ec7edec9f853b95b0f1d7782599f34f80dfe34df91870b7f43711aca6424eefd
Oct 31 03:28:24.535596 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:24.535561    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq" event={"ID":"41c075e6-b57b-4e72-ad58-59450e64c8a3","Type":"ContainerStarted","Data":"ec7edec9f853b95b0f1d7782599f34f80dfe34df91870b7f43711aca6424eefd"}
Oct 31 03:28:24.540358 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:24.540341    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv"
Oct 31 03:28:26.683970 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:26.683933    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"
Oct 31 03:28:26.698671 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:26.698631    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"
Oct 31 03:28:26.752808 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:26.752769    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"]
Oct 31 03:28:26.757396 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:26.757374    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"
Oct 31 03:28:26.812149 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:26.812121    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 31 03:28:26.816812 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:26.816777    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"]
Oct 31 03:28:26.833828 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:26.833796    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5\" (UID: \"5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"
Oct 31 03:28:26.833828 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:26.833830    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5\" (UID: \"5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"
Oct 31 03:28:26.834032 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:26.833978    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5\" (UID: \"5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"
Oct 31 03:28:26.834032 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:26.834027    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5\" (UID: \"5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"
Oct 31 03:28:26.834095 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:26.834054    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmhmm\" (UniqueName: \"kubernetes.io/projected/5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7-kube-api-access-xmhmm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5\" (UID: \"5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"
Oct 31 03:28:26.834218 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:26.834206    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5\" (UID: \"5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"
Oct 31 03:28:26.935623 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:26.935541    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5\" (UID: \"5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"
Oct 31 03:28:26.935623 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:26.935599    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5\" (UID: \"5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"
Oct 31 03:28:26.935623 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:26.935618    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5\" (UID: \"5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"
Oct 31 03:28:26.935843 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:26.935657    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5\" (UID: \"5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"
Oct 31 03:28:26.935843 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:26.935675    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5\" (UID: \"5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"
Oct 31 03:28:26.935843 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:26.935810    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmhmm\" (UniqueName: \"kubernetes.io/projected/5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7-kube-api-access-xmhmm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5\" (UID: \"5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"
Oct 31 03:28:26.936010 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:26.935986    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5\" (UID: \"5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"
Oct 31 03:28:26.936053 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:26.936033    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5\" (UID: \"5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"
Oct 31 03:28:26.936324 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:26.936302    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5\" (UID: \"5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"
Oct 31 03:28:26.938227 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:26.938195    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5\" (UID: \"5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"
Oct 31 03:28:26.938440 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:26.938421    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5\" (UID: \"5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"
Oct 31 03:28:26.997451 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:26.997416    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmhmm\" (UniqueName: \"kubernetes.io/projected/5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7-kube-api-access-xmhmm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5\" (UID: \"5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"
Oct 31 03:28:27.075328 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:27.075291    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"
Oct 31 03:28:27.457439 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:27.457413    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"]
Oct 31 03:28:27.459124 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:28:27.459100    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c66fbd6_0415_4cb4_a9ca_fdfc437c55f7.slice/crio-c74952662278504cb6ea05769e1fb8b4dd38b28ea2a1f028648dfcbd2e513ddb WatchSource:0}: Error finding container c74952662278504cb6ea05769e1fb8b4dd38b28ea2a1f028648dfcbd2e513ddb: Status 404 returned error can't find the container with id c74952662278504cb6ea05769e1fb8b4dd38b28ea2a1f028648dfcbd2e513ddb
Oct 31 03:28:27.557693 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:27.557660    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq" event={"ID":"41c075e6-b57b-4e72-ad58-59450e64c8a3","Type":"ContainerStarted","Data":"eed5ff92dc33a09d16b5d414be29710bb6b8648866d92b11eaa9f6d0bbf7e616"}
Oct 31 03:28:27.558897 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:27.558867    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5" event={"ID":"5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7","Type":"ContainerStarted","Data":"c74952662278504cb6ea05769e1fb8b4dd38b28ea2a1f028648dfcbd2e513ddb"}
Oct 31 03:28:27.569801 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:27.569781    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dsdwd"
Oct 31 03:28:27.675935 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:27.675862    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq" podStartSLOduration=1.981189555 podStartE2EDuration="4.675847477s" totalImagesPullingTime="2.694657922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:27.674698575 +0000 UTC m=+533.185680954" watchObservedRunningTime="2026-10-31 03:28:27.675847477 +0000 UTC m=+533.186829853"
Oct 31 03:28:27.752970 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:27.752876    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq"
Oct 31 03:28:28.567120 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:28.567073    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5" event={"ID":"5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7","Type":"ContainerStarted","Data":"d22ce52082b81e38002a44cc9a4a76a8d6ff83d1ecaa813eb9189df1ed4f817f"}
Oct 31 03:28:28.687253 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:28.687195    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5" podStartSLOduration=2.09679166 podStartE2EDuration="2.687180527s" totalImagesPullingTime="590.388867ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:28.684979907 +0000 UTC m=+534.195962284" watchObservedRunningTime="2026-10-31 03:28:28.687180527 +0000 UTC m=+534.198162903"
Oct 31 03:28:29.076135 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:29.076106    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"
Oct 31 03:28:29.984821 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:29.984788    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"
Oct 31 03:28:29.997170 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:29.997142    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"
Oct 31 03:28:30.587863 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:30.587829    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r"
Oct 31 03:28:33.752928 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:33.752873    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq"
Oct 31 03:28:33.765118 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:33.765093    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq"
Oct 31 03:28:34.611029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:34.611005    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq"
Oct 31 03:28:35.865974 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:35.865941    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"]
Oct 31 03:28:35.869957 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:35.869938    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"
Oct 31 03:28:35.936669 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:35.936632    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 31 03:28:35.974505 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:35.974473    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"]
Oct 31 03:28:36.030421 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:36.030381    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94dq4\" (UniqueName: \"kubernetes.io/projected/2b7caa8c-a0aa-4fbd-8918-288e39fc4f56-kube-api-access-94dq4\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb\" (UID: \"2b7caa8c-a0aa-4fbd-8918-288e39fc4f56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"
Oct 31 03:28:36.030593 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:36.030443    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b7caa8c-a0aa-4fbd-8918-288e39fc4f56-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb\" (UID: \"2b7caa8c-a0aa-4fbd-8918-288e39fc4f56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"
Oct 31 03:28:36.030593 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:36.030537    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b7caa8c-a0aa-4fbd-8918-288e39fc4f56-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb\" (UID: \"2b7caa8c-a0aa-4fbd-8918-288e39fc4f56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"
Oct 31 03:28:36.030667 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:36.030650    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7caa8c-a0aa-4fbd-8918-288e39fc4f56-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb\" (UID: \"2b7caa8c-a0aa-4fbd-8918-288e39fc4f56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"
Oct 31 03:28:36.030710 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:36.030697    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b7caa8c-a0aa-4fbd-8918-288e39fc4f56-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb\" (UID: \"2b7caa8c-a0aa-4fbd-8918-288e39fc4f56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"
Oct 31 03:28:36.030744 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:36.030715    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b7caa8c-a0aa-4fbd-8918-288e39fc4f56-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb\" (UID: \"2b7caa8c-a0aa-4fbd-8918-288e39fc4f56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"
Oct 31 03:28:36.131428 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:36.131356    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7caa8c-a0aa-4fbd-8918-288e39fc4f56-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb\" (UID: \"2b7caa8c-a0aa-4fbd-8918-288e39fc4f56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"
Oct 31 03:28:36.131428 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:36.131406    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b7caa8c-a0aa-4fbd-8918-288e39fc4f56-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb\" (UID: \"2b7caa8c-a0aa-4fbd-8918-288e39fc4f56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"
Oct 31 03:28:36.131428 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:36.131429    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b7caa8c-a0aa-4fbd-8918-288e39fc4f56-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb\" (UID: \"2b7caa8c-a0aa-4fbd-8918-288e39fc4f56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"
Oct 31 03:28:36.131688 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:36.131500    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94dq4\" (UniqueName: \"kubernetes.io/projected/2b7caa8c-a0aa-4fbd-8918-288e39fc4f56-kube-api-access-94dq4\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb\" (UID: \"2b7caa8c-a0aa-4fbd-8918-288e39fc4f56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"
Oct 31 03:28:36.131688 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:36.131554    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b7caa8c-a0aa-4fbd-8918-288e39fc4f56-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb\" (UID: \"2b7caa8c-a0aa-4fbd-8918-288e39fc4f56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"
Oct 31 03:28:36.131688 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:36.131582    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b7caa8c-a0aa-4fbd-8918-288e39fc4f56-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb\" (UID: \"2b7caa8c-a0aa-4fbd-8918-288e39fc4f56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"
Oct 31 03:28:36.131942 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:36.131909    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b7caa8c-a0aa-4fbd-8918-288e39fc4f56-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb\" (UID: \"2b7caa8c-a0aa-4fbd-8918-288e39fc4f56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"
Oct 31 03:28:36.132050 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:36.131966    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b7caa8c-a0aa-4fbd-8918-288e39fc4f56-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb\" (UID: \"2b7caa8c-a0aa-4fbd-8918-288e39fc4f56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"
Oct 31 03:28:36.132050 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:36.131987    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b7caa8c-a0aa-4fbd-8918-288e39fc4f56-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb\" (UID: \"2b7caa8c-a0aa-4fbd-8918-288e39fc4f56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"
Oct 31 03:28:36.133918 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:36.133879    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b7caa8c-a0aa-4fbd-8918-288e39fc4f56-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb\" (UID: \"2b7caa8c-a0aa-4fbd-8918-288e39fc4f56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"
Oct 31 03:28:36.134039 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:36.133983    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7caa8c-a0aa-4fbd-8918-288e39fc4f56-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb\" (UID: \"2b7caa8c-a0aa-4fbd-8918-288e39fc4f56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"
Oct 31 03:28:36.193103 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:36.193078    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94dq4\" (UniqueName: \"kubernetes.io/projected/2b7caa8c-a0aa-4fbd-8918-288e39fc4f56-kube-api-access-94dq4\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb\" (UID: \"2b7caa8c-a0aa-4fbd-8918-288e39fc4f56\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"
Oct 31 03:28:36.483242 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:36.483215    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"
Oct 31 03:28:36.862589 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:36.862561    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"]
Oct 31 03:28:36.864697 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:28:36.864666    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7caa8c_a0aa_4fbd_8918_288e39fc4f56.slice/crio-2d3018ff3a28d0916d14de6814f813097760817fb56a40fc883ba339ccbe61f4 WatchSource:0}: Error finding container 2d3018ff3a28d0916d14de6814f813097760817fb56a40fc883ba339ccbe61f4: Status 404 returned error can't find the container with id 2d3018ff3a28d0916d14de6814f813097760817fb56a40fc883ba339ccbe61f4
Oct 31 03:28:37.075921 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:37.075873    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"
Oct 31 03:28:37.088875 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:37.088853    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"
Oct 31 03:28:37.619653 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:37.619617    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb" event={"ID":"2b7caa8c-a0aa-4fbd-8918-288e39fc4f56","Type":"ContainerStarted","Data":"f22be9a1c31c3984df5c1064038f41c473302d78768260005735f391eef993a2"}
Oct 31 03:28:37.619953 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:37.619681    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb" event={"ID":"2b7caa8c-a0aa-4fbd-8918-288e39fc4f56","Type":"ContainerStarted","Data":"2d3018ff3a28d0916d14de6814f813097760817fb56a40fc883ba339ccbe61f4"}
Oct 31 03:28:37.637393 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:37.637350    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5"
Oct 31 03:28:37.739437 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:37.739382    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb" podStartSLOduration=2.168471669 podStartE2EDuration="2.739367479s" totalImagesPullingTime="570.89581ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:37.738370948 +0000 UTC m=+543.249353337" watchObservedRunningTime="2026-10-31 03:28:37.739367479 +0000 UTC m=+543.250349902"
Oct 31 03:28:38.484411 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:38.484379    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"
Oct 31 03:28:46.483878 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:46.483838    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"
Oct 31 03:28:46.497008 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:46.496980    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"
Oct 31 03:28:46.679119 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:46.679094    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb"
Oct 31 03:28:49.001216 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:28:49.001175    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:28:59.623697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.623665    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"]
Oct 31 03:28:59.632012 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.631991    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.774054 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.774023    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.774054 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.774056    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.774259 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.774088    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a944162-110a-4337-a053-21947af7dca1-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.774308 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.774257    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a944162-110a-4337-a053-21947af7dca1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.774355 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.774334    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.774409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.774369    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.774460 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.774439    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.774525 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.774507    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kj4z\" (UniqueName: \"kubernetes.io/projected/3a944162-110a-4337-a053-21947af7dca1-kube-api-access-2kj4z\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.774573 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.774546    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a944162-110a-4337-a053-21947af7dca1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.843727 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.843686    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"]
Oct 31 03:28:59.875411 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.875322    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.875411 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.875363    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.875411 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.875391    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a944162-110a-4337-a053-21947af7dca1-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.875695 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.875418    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a944162-110a-4337-a053-21947af7dca1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.875695 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.875446    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.875695 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.875465    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.875695 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.875484    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.875695 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.875505    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kj4z\" (UniqueName: \"kubernetes.io/projected/3a944162-110a-4337-a053-21947af7dca1-kube-api-access-2kj4z\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.875695 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.875532    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a944162-110a-4337-a053-21947af7dca1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.876049 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.875771    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.876049 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.875928    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.876049 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.875978    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.876049 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.876015    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.876440 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.876414    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a944162-110a-4337-a053-21947af7dca1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.877953 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.877928    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.878149 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.878132    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a944162-110a-4337-a053-21947af7dca1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.937176 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.937137    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a944162-110a-4337-a053-21947af7dca1-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.937324 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.937213    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kj4z\" (UniqueName: \"kubernetes.io/projected/3a944162-110a-4337-a053-21947af7dca1-kube-api-access-2kj4z\") pod \"maas-default-gateway-openshift-default-fd4b99879-jncxp\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:28:59.945135 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:28:59.945104    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:29:00.422916 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:00.422872    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"]
Oct 31 03:29:00.424017 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:29:00.423991    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a944162_110a_4337_a053_21947af7dca1.slice/crio-4ba6f729a36061427162f6f1f4eec862610fb31647fe7763cf593e456a99cdca WatchSource:0}: Error finding container 4ba6f729a36061427162f6f1f4eec862610fb31647fe7763cf593e456a99cdca: Status 404 returned error can't find the container with id 4ba6f729a36061427162f6f1f4eec862610fb31647fe7763cf593e456a99cdca
Oct 31 03:29:00.426279 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:00.426253    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:29:00.426368 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:00.426314    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:29:00.426368 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:00.426341    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:29:00.747311 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:00.747277    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp" event={"ID":"3a944162-110a-4337-a053-21947af7dca1","Type":"ContainerStarted","Data":"5007d072c6844b24e2d205ae46d9e9bfcecd86c76b7ec46c7c0d5c8571f10e00"}
Oct 31 03:29:00.747711 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:00.747328    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp" event={"ID":"3a944162-110a-4337-a053-21947af7dca1","Type":"ContainerStarted","Data":"4ba6f729a36061427162f6f1f4eec862610fb31647fe7763cf593e456a99cdca"}
Oct 31 03:29:00.871577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:00.871523    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp" podStartSLOduration=1.871506236 podStartE2EDuration="1.871506236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:00.869424847 +0000 UTC m=+566.380407226" watchObservedRunningTime="2026-10-31 03:29:00.871506236 +0000 UTC m=+566.382488614"
Oct 31 03:29:00.945466 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:00.945429    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:29:01.319665 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:01.319625    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-nzlnr"]
Oct 31 03:29:01.323689 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:01.323674    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-nzlnr"
Oct 31 03:29:01.401035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:01.401004    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 31 03:29:01.403186 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:01.403153    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 31 03:29:01.403186 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:01.401869    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b75c511-da96-4c85-b111-4e99659634f3-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-nzlnr\" (UID: \"1b75c511-da96-4c85-b111-4e99659634f3\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-nzlnr"
Oct 31 03:29:01.403395 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:01.403228    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd8cw\" (UniqueName: \"kubernetes.io/projected/1b75c511-da96-4c85-b111-4e99659634f3-kube-api-access-cd8cw\") pod \"ai-gateway-controller-5d9cd99f45-nzlnr\" (UID: \"1b75c511-da96-4c85-b111-4e99659634f3\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-nzlnr"
Oct 31 03:29:01.403395 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:01.402081    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-49h29\""
Oct 31 03:29:01.405911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:01.405856    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-nzlnr"]
Oct 31 03:29:01.504782 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:01.504748    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b75c511-da96-4c85-b111-4e99659634f3-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-nzlnr\" (UID: \"1b75c511-da96-4c85-b111-4e99659634f3\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-nzlnr"
Oct 31 03:29:01.505108 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:01.505074    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd8cw\" (UniqueName: \"kubernetes.io/projected/1b75c511-da96-4c85-b111-4e99659634f3-kube-api-access-cd8cw\") pod \"ai-gateway-controller-5d9cd99f45-nzlnr\" (UID: \"1b75c511-da96-4c85-b111-4e99659634f3\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-nzlnr"
Oct 31 03:29:01.508404 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:01.508382    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b75c511-da96-4c85-b111-4e99659634f3-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-nzlnr\" (UID: \"1b75c511-da96-4c85-b111-4e99659634f3\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-nzlnr"
Oct 31 03:29:01.567107 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:01.567073    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd8cw\" (UniqueName: \"kubernetes.io/projected/1b75c511-da96-4c85-b111-4e99659634f3-kube-api-access-cd8cw\") pod \"ai-gateway-controller-5d9cd99f45-nzlnr\" (UID: \"1b75c511-da96-4c85-b111-4e99659634f3\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-nzlnr"
Oct 31 03:29:01.637408 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:01.637316    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-nzlnr"
Oct 31 03:29:01.946911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:01.946819    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jncxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 31 03:29:01.946911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:01.946869    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp" podUID="3a944162-110a-4337-a053-21947af7dca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 31 03:29:02.034464 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:02.034437    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-nzlnr"]
Oct 31 03:29:02.035368 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:29:02.035344    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b75c511_da96_4c85_b111_4e99659634f3.slice/crio-656aa392c33da677bbc08e77093f4e78874be11ef2352079b40696704239c2c8 WatchSource:0}: Error finding container 656aa392c33da677bbc08e77093f4e78874be11ef2352079b40696704239c2c8: Status 404 returned error can't find the container with id 656aa392c33da677bbc08e77093f4e78874be11ef2352079b40696704239c2c8
Oct 31 03:29:02.759728 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:02.759692    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-nzlnr" event={"ID":"1b75c511-da96-4c85-b111-4e99659634f3","Type":"ContainerStarted","Data":"656aa392c33da677bbc08e77093f4e78874be11ef2352079b40696704239c2c8"}
Oct 31 03:29:02.945918 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:02.945852    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jncxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 31 03:29:02.946117 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:02.945945    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp" podUID="3a944162-110a-4337-a053-21947af7dca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 31 03:29:03.946254 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:03.946216    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jncxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 31 03:29:03.946623 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:03.946276    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp" podUID="3a944162-110a-4337-a053-21947af7dca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 31 03:29:04.945746 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:04.945714    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jncxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 31 03:29:04.945973 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:04.945768    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp" podUID="3a944162-110a-4337-a053-21947af7dca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 31 03:29:05.945954 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:05.945918    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jncxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 31 03:29:05.946381 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:05.945975    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp" podUID="3a944162-110a-4337-a053-21947af7dca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 31 03:29:06.945465 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:06.945440    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jncxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 31 03:29:06.945579 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:06.945482    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp" podUID="3a944162-110a-4337-a053-21947af7dca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 31 03:29:07.815386 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:07.815333    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-nzlnr" event={"ID":"1b75c511-da96-4c85-b111-4e99659634f3","Type":"ContainerStarted","Data":"d50183264f9cb974f5e2132539d6cd480584e096a152a24f8f49999d3f789e18"}
Oct 31 03:29:07.816948 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:07.816903    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d9cd99f45-nzlnr"
Oct 31 03:29:07.933349 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:07.933286    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d9cd99f45-nzlnr" podStartSLOduration=2.040784091 podStartE2EDuration="6.933271549s" totalImagesPullingTime="4.892487458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:07.932105732 +0000 UTC m=+573.443088114" watchObservedRunningTime="2026-10-31 03:29:07.933271549 +0000 UTC m=+573.444253925"
Oct 31 03:29:08.568412 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:08.568377    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:29:08.568955 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:08.568935    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:29:08.570567 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:08.570549    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:29:11.463340 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:11.463303    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-2z9q8"]
Oct 31 03:29:11.468273 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:11.468251    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-2z9q8"
Oct 31 03:29:11.522351 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:11.522315    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 31 03:29:11.572175 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:11.572141    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-mp6g4\""
Oct 31 03:29:11.574929 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:11.574905    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-2z9q8"]
Oct 31 03:29:11.614160 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:11.614120    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhjrw\" (UniqueName: \"kubernetes.io/projected/40e3764e-c5f1-4c75-b49e-13d815a622d2-kube-api-access-fhjrw\") pod \"payload-processing-55fbbbf9dc-2z9q8\" (UID: \"40e3764e-c5f1-4c75-b49e-13d815a622d2\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-2z9q8"
Oct 31 03:29:11.614322 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:11.614263    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e3764e-c5f1-4c75-b49e-13d815a622d2-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-2z9q8\" (UID: \"40e3764e-c5f1-4c75-b49e-13d815a622d2\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-2z9q8"
Oct 31 03:29:11.715470 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:11.715381    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhjrw\" (UniqueName: \"kubernetes.io/projected/40e3764e-c5f1-4c75-b49e-13d815a622d2-kube-api-access-fhjrw\") pod \"payload-processing-55fbbbf9dc-2z9q8\" (UID: \"40e3764e-c5f1-4c75-b49e-13d815a622d2\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-2z9q8"
Oct 31 03:29:11.715618 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:11.715502    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e3764e-c5f1-4c75-b49e-13d815a622d2-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-2z9q8\" (UID: \"40e3764e-c5f1-4c75-b49e-13d815a622d2\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-2z9q8"
Oct 31 03:29:11.716297 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:11.716276    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e3764e-c5f1-4c75-b49e-13d815a622d2-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-2z9q8\" (UID: \"40e3764e-c5f1-4c75-b49e-13d815a622d2\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-2z9q8"
Oct 31 03:29:11.776851 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:11.776821    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhjrw\" (UniqueName: \"kubernetes.io/projected/40e3764e-c5f1-4c75-b49e-13d815a622d2-kube-api-access-fhjrw\") pod \"payload-processing-55fbbbf9dc-2z9q8\" (UID: \"40e3764e-c5f1-4c75-b49e-13d815a622d2\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-2z9q8"
Oct 31 03:29:11.784146 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:11.784122    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-2z9q8"
Oct 31 03:29:12.164327 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:12.164291    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-2z9q8"]
Oct 31 03:29:12.167926 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:29:12.167867    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e3764e_c5f1_4c75_b49e_13d815a622d2.slice/crio-5e07c9f3de1e7bb2e407087ee02e0d1ddd0656729f2a21d231294ae520c9a823 WatchSource:0}: Error finding container 5e07c9f3de1e7bb2e407087ee02e0d1ddd0656729f2a21d231294ae520c9a823: Status 404 returned error can't find the container with id 5e07c9f3de1e7bb2e407087ee02e0d1ddd0656729f2a21d231294ae520c9a823
Oct 31 03:29:12.847261 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:12.847227    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-2z9q8" event={"ID":"40e3764e-c5f1-4c75-b49e-13d815a622d2","Type":"ContainerStarted","Data":"5e07c9f3de1e7bb2e407087ee02e0d1ddd0656729f2a21d231294ae520c9a823"}
Oct 31 03:29:14.860998 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:14.860961    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-2z9q8" event={"ID":"40e3764e-c5f1-4c75-b49e-13d815a622d2","Type":"ContainerStarted","Data":"c6d4eb6fe4b24dde404041ff1f2c78a7a8e43331c6ab10f9847f95dbe49bbd3f"}
Oct 31 03:29:14.861392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:14.861048    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-2z9q8"
Oct 31 03:29:14.979681 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:14.979571    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-2z9q8" podStartSLOduration=1.423029114 podStartE2EDuration="3.979557825s" totalImagesPullingTime="2.556528711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:14.977435962 +0000 UTC m=+580.488418338" watchObservedRunningTime="2026-10-31 03:29:14.979557825 +0000 UTC m=+580.490540201"
Oct 31 03:29:18.823699 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:18.823627    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d9cd99f45-nzlnr"
Oct 31 03:29:19.004980 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:29:19.004936    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:29:19.198735 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:19.198644    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-66b48bb959-2vdzd"]
Oct 31 03:29:19.202524 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:19.202508    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66b48bb959-2vdzd"
Oct 31 03:29:19.261983 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:19.261943    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-66b48bb959-2vdzd"]
Oct 31 03:29:19.297798 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:19.297756    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnw8d\" (UniqueName: \"kubernetes.io/projected/1267ec47-4190-45eb-bd44-2850e7af98e0-kube-api-access-xnw8d\") pod \"ai-gateway-controller-66b48bb959-2vdzd\" (UID: \"1267ec47-4190-45eb-bd44-2850e7af98e0\") " pod="opendatahub/ai-gateway-controller-66b48bb959-2vdzd"
Oct 31 03:29:19.298004 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:19.297812    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1267ec47-4190-45eb-bd44-2850e7af98e0-metrics-certs\") pod \"ai-gateway-controller-66b48bb959-2vdzd\" (UID: \"1267ec47-4190-45eb-bd44-2850e7af98e0\") " pod="opendatahub/ai-gateway-controller-66b48bb959-2vdzd"
Oct 31 03:29:19.399031 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:19.398993    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnw8d\" (UniqueName: \"kubernetes.io/projected/1267ec47-4190-45eb-bd44-2850e7af98e0-kube-api-access-xnw8d\") pod \"ai-gateway-controller-66b48bb959-2vdzd\" (UID: \"1267ec47-4190-45eb-bd44-2850e7af98e0\") " pod="opendatahub/ai-gateway-controller-66b48bb959-2vdzd"
Oct 31 03:29:19.399031 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:19.399043    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1267ec47-4190-45eb-bd44-2850e7af98e0-metrics-certs\") pod \"ai-gateway-controller-66b48bb959-2vdzd\" (UID: \"1267ec47-4190-45eb-bd44-2850e7af98e0\") " pod="opendatahub/ai-gateway-controller-66b48bb959-2vdzd"
Oct 31 03:29:19.401599 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:19.401578    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1267ec47-4190-45eb-bd44-2850e7af98e0-metrics-certs\") pod \"ai-gateway-controller-66b48bb959-2vdzd\" (UID: \"1267ec47-4190-45eb-bd44-2850e7af98e0\") " pod="opendatahub/ai-gateway-controller-66b48bb959-2vdzd"
Oct 31 03:29:19.462009 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:19.461976    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnw8d\" (UniqueName: \"kubernetes.io/projected/1267ec47-4190-45eb-bd44-2850e7af98e0-kube-api-access-xnw8d\") pod \"ai-gateway-controller-66b48bb959-2vdzd\" (UID: \"1267ec47-4190-45eb-bd44-2850e7af98e0\") " pod="opendatahub/ai-gateway-controller-66b48bb959-2vdzd"
Oct 31 03:29:19.516093 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:19.516061    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66b48bb959-2vdzd"
Oct 31 03:29:19.894874 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:19.894843    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-66b48bb959-2vdzd"]
Oct 31 03:29:19.895317 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:29:19.894876    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1267ec47_4190_45eb_bd44_2850e7af98e0.slice/crio-13fe50d9bc8e1483fa023f7b0d06c832fde9c8f75326ebbf75e9caa8f6adf381 WatchSource:0}: Error finding container 13fe50d9bc8e1483fa023f7b0d06c832fde9c8f75326ebbf75e9caa8f6adf381: Status 404 returned error can't find the container with id 13fe50d9bc8e1483fa023f7b0d06c832fde9c8f75326ebbf75e9caa8f6adf381
Oct 31 03:29:20.895745 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:20.895708    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66b48bb959-2vdzd" event={"ID":"1267ec47-4190-45eb-bd44-2850e7af98e0","Type":"ContainerStarted","Data":"285ec81c6bec8f26f0359eb79c35e5bb5ec22301bb45cd13228e713f1fd3780d"}
Oct 31 03:29:20.896179 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:20.895812    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66b48bb959-2vdzd" event={"ID":"1267ec47-4190-45eb-bd44-2850e7af98e0","Type":"ContainerStarted","Data":"13fe50d9bc8e1483fa023f7b0d06c832fde9c8f75326ebbf75e9caa8f6adf381"}
Oct 31 03:29:20.896179 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:20.895986    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-66b48bb959-2vdzd"
Oct 31 03:29:21.016366 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:21.016266    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-66b48bb959-2vdzd" podStartSLOduration=1.190742841 podStartE2EDuration="2.016252686s" totalImagesPullingTime="825.509845ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:21.013698358 +0000 UTC m=+586.524680735" watchObservedRunningTime="2026-10-31 03:29:21.016252686 +0000 UTC m=+586.527235063"
Oct 31 03:29:25.869695 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:25.869666    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-2z9q8"
Oct 31 03:29:31.903941 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:31.903878    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-66b48bb959-2vdzd"
Oct 31 03:29:32.074801 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:32.074770    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-nzlnr"]
Oct 31 03:29:32.075008 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:32.074987    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d9cd99f45-nzlnr" podUID="1b75c511-da96-4c85-b111-4e99659634f3" containerName="manager" containerID="cri-o://d50183264f9cb974f5e2132539d6cd480584e096a152a24f8f49999d3f789e18" gracePeriod=30
Oct 31 03:29:32.336587 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:32.336561    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-nzlnr"
Oct 31 03:29:32.432948 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:32.432910    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b75c511-da96-4c85-b111-4e99659634f3-kube-api-access-cd8cw\" (UniqueName: \"kubernetes.io/projected/1b75c511-da96-4c85-b111-4e99659634f3-kube-api-access-cd8cw\") pod \"1b75c511-da96-4c85-b111-4e99659634f3\" (UID: \"1b75c511-da96-4c85-b111-4e99659634f3\") "
Oct 31 03:29:32.433147 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:32.433018    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b75c511-da96-4c85-b111-4e99659634f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b75c511-da96-4c85-b111-4e99659634f3-metrics-certs\") pod \"1b75c511-da96-4c85-b111-4e99659634f3\" (UID: \"1b75c511-da96-4c85-b111-4e99659634f3\") "
Oct 31 03:29:32.435205 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:32.435160    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b75c511-da96-4c85-b111-4e99659634f3-metrics-certs" pod "1b75c511-da96-4c85-b111-4e99659634f3" (UID: "1b75c511-da96-4c85-b111-4e99659634f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:29:32.435306 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:32.435215    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b75c511-da96-4c85-b111-4e99659634f3-kube-api-access-cd8cw" pod "1b75c511-da96-4c85-b111-4e99659634f3" (UID: "1b75c511-da96-4c85-b111-4e99659634f3"). InnerVolumeSpecName "kube-api-access-cd8cw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:29:32.533844 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:32.533805    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b75c511-da96-4c85-b111-4e99659634f3-metrics-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:29:32.533844 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:32.533839    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd8cw\" (UniqueName: \"kubernetes.io/projected/1b75c511-da96-4c85-b111-4e99659634f3-kube-api-access-cd8cw\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:29:32.963368 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:32.963334    4447 generic.go:356] "Generic (PLEG): container finished" podID="1b75c511-da96-4c85-b111-4e99659634f3" containerID="d50183264f9cb974f5e2132539d6cd480584e096a152a24f8f49999d3f789e18" exitCode=0
Oct 31 03:29:32.963368 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:32.963373    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-nzlnr" event={"ID":"1b75c511-da96-4c85-b111-4e99659634f3","Type":"ContainerDied","Data":"d50183264f9cb974f5e2132539d6cd480584e096a152a24f8f49999d3f789e18"}
Oct 31 03:29:32.964012 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:32.963391    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-nzlnr"
Oct 31 03:29:32.964012 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:32.963415    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-nzlnr" event={"ID":"1b75c511-da96-4c85-b111-4e99659634f3","Type":"ContainerDied","Data":"656aa392c33da677bbc08e77093f4e78874be11ef2352079b40696704239c2c8"}
Oct 31 03:29:32.964012 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:32.963436    4447 scope.go:122] "RemoveContainer" containerID="d50183264f9cb974f5e2132539d6cd480584e096a152a24f8f49999d3f789e18"
Oct 31 03:29:32.972605 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:32.972579    4447 scope.go:122] "RemoveContainer" containerID="d50183264f9cb974f5e2132539d6cd480584e096a152a24f8f49999d3f789e18"
Oct 31 03:29:32.972929 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:29:32.972878    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d50183264f9cb974f5e2132539d6cd480584e096a152a24f8f49999d3f789e18\": container with ID starting with d50183264f9cb974f5e2132539d6cd480584e096a152a24f8f49999d3f789e18 not found: ID does not exist" containerID="d50183264f9cb974f5e2132539d6cd480584e096a152a24f8f49999d3f789e18"
Oct 31 03:29:32.973016 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:32.972945    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d50183264f9cb974f5e2132539d6cd480584e096a152a24f8f49999d3f789e18"} err="failed to get container status \"d50183264f9cb974f5e2132539d6cd480584e096a152a24f8f49999d3f789e18\": rpc error: code = NotFound desc = could not find container \"d50183264f9cb974f5e2132539d6cd480584e096a152a24f8f49999d3f789e18\": container with ID starting with d50183264f9cb974f5e2132539d6cd480584e096a152a24f8f49999d3f789e18 not found: ID does not exist"
Oct 31 03:29:33.129721 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:33.129681    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-nzlnr"]
Oct 31 03:29:33.179551 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:33.179517    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-nzlnr"]
Oct 31 03:29:34.607759 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:34.607728    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b75c511-da96-4c85-b111-4e99659634f3" path="/var/lib/kubelet/pods/1b75c511-da96-4c85-b111-4e99659634f3/volumes"
Oct 31 03:29:34.688313 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:34.688279    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 03:29:34.692706 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:29:34.692676    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 03:29:49.004780 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:29:49.004740    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:30:00.179575 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:00.179531    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t"]
Oct 31 03:30:00.180297 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:00.180277    4447 cpu_manager.go:414] "containerMap: removing container" podUID="1b75c511-da96-4c85-b111-4e99659634f3" containerName="manager"
Oct 31 03:30:00.180393 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:00.180299    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b75c511-da96-4c85-b111-4e99659634f3" containerName="manager"
Oct 31 03:30:00.180445 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:00.180420    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b75c511-da96-4c85-b111-4e99659634f3" containerName="manager"
Oct 31 03:30:00.183694 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:00.183673    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t"
Oct 31 03:30:00.218298 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:00.218264    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b24b7da8-1429-4d60-b111-cc66514b84c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890290-jz68t\" (UID: \"b24b7da8-1429-4d60-b111-cc66514b84c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t"
Oct 31 03:30:00.218298 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:00.218299    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9r9g\" (UniqueName: \"kubernetes.io/projected/b24b7da8-1429-4d60-b111-cc66514b84c9-kube-api-access-q9r9g\") pod \"maas-api-key-cleanup-29890290-jz68t\" (UID: \"b24b7da8-1429-4d60-b111-cc66514b84c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t"
Oct 31 03:30:00.237861 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:00.237829    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 03:30:00.278439 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:00.278406    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t"]
Oct 31 03:30:00.319661 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:00.319630    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b24b7da8-1429-4d60-b111-cc66514b84c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890290-jz68t\" (UID: \"b24b7da8-1429-4d60-b111-cc66514b84c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t"
Oct 31 03:30:00.319661 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:00.319665    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9r9g\" (UniqueName: \"kubernetes.io/projected/b24b7da8-1429-4d60-b111-cc66514b84c9-kube-api-access-q9r9g\") pod \"maas-api-key-cleanup-29890290-jz68t\" (UID: \"b24b7da8-1429-4d60-b111-cc66514b84c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t"
Oct 31 03:30:00.320299 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:00.320281    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b24b7da8-1429-4d60-b111-cc66514b84c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890290-jz68t\" (UID: \"b24b7da8-1429-4d60-b111-cc66514b84c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t"
Oct 31 03:30:00.381745 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:00.381704    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9r9g\" (UniqueName: \"kubernetes.io/projected/b24b7da8-1429-4d60-b111-cc66514b84c9-kube-api-access-q9r9g\") pod \"maas-api-key-cleanup-29890290-jz68t\" (UID: \"b24b7da8-1429-4d60-b111-cc66514b84c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t"
Oct 31 03:30:00.496621 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:00.496584    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t"
Oct 31 03:30:00.869823 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:00.869786    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t"]
Oct 31 03:30:00.871533 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:30:00.871513    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24b7da8_1429_4d60_b111_cc66514b84c9.slice/crio-30b54183ba38c8b5a6be16a81d3e0aa31749b4611d0021870fa1f6e1382ca34d WatchSource:0}: Error finding container 30b54183ba38c8b5a6be16a81d3e0aa31749b4611d0021870fa1f6e1382ca34d: Status 404 returned error can't find the container with id 30b54183ba38c8b5a6be16a81d3e0aa31749b4611d0021870fa1f6e1382ca34d
Oct 31 03:30:00.873705 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:00.873689    4447 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:30:01.139829 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:01.139741    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t" event={"ID":"b24b7da8-1429-4d60-b111-cc66514b84c9","Type":"ContainerStarted","Data":"30b54183ba38c8b5a6be16a81d3e0aa31749b4611d0021870fa1f6e1382ca34d"}
Oct 31 03:30:10.193940 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:10.193906    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t" event={"ID":"b24b7da8-1429-4d60-b111-cc66514b84c9","Type":"ContainerStarted","Data":"8a9b8e9933e51d9fb6602ff71171c24a7d0846e20ed5fee4dae2094196c5fbd7"}
Oct 31 03:30:10.309767 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:10.309667    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t" podStartSLOduration=1.461313494 podStartE2EDuration="10.309654361s" totalImagesPullingTime="8.848340867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:10.309345626 +0000 UTC m=+635.820328015" watchObservedRunningTime="2026-10-31 03:30:10.309654361 +0000 UTC m=+635.820636737"
Oct 31 03:30:11.201570 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:11.201539    4447 generic.go:356] "Generic (PLEG): container finished" podID="b24b7da8-1429-4d60-b111-cc66514b84c9" containerID="8a9b8e9933e51d9fb6602ff71171c24a7d0846e20ed5fee4dae2094196c5fbd7" exitCode=60
Oct 31 03:30:11.202030 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:11.201631    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t" event={"ID":"b24b7da8-1429-4d60-b111-cc66514b84c9","Type":"ContainerDied","Data":"8a9b8e9933e51d9fb6602ff71171c24a7d0846e20ed5fee4dae2094196c5fbd7"}
Oct 31 03:30:11.202030 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:11.201825    4447 scope.go:122] "RemoveContainer" containerID="8a9b8e9933e51d9fb6602ff71171c24a7d0846e20ed5fee4dae2094196c5fbd7"
Oct 31 03:30:12.209276 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:12.209244    4447 generic.go:356] "Generic (PLEG): container finished" podID="b24b7da8-1429-4d60-b111-cc66514b84c9" containerID="32a2f90599d20871f953ea9a7ad91efebf07f3ffcbab9b2f158afe616a7494ce" exitCode=60
Oct 31 03:30:12.209744 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:12.209312    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t" event={"ID":"b24b7da8-1429-4d60-b111-cc66514b84c9","Type":"ContainerDied","Data":"32a2f90599d20871f953ea9a7ad91efebf07f3ffcbab9b2f158afe616a7494ce"}
Oct 31 03:30:12.209744 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:12.209371    4447 scope.go:122] "RemoveContainer" containerID="8a9b8e9933e51d9fb6602ff71171c24a7d0846e20ed5fee4dae2094196c5fbd7"
Oct 31 03:30:12.209744 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:12.209619    4447 scope.go:122] "RemoveContainer" containerID="32a2f90599d20871f953ea9a7ad91efebf07f3ffcbab9b2f158afe616a7494ce"
Oct 31 03:30:12.209946 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:30:12.209924    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890290-jz68t_odh-ai-gateway-infra(b24b7da8-1429-4d60-b111-cc66514b84c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t" podUID="b24b7da8-1429-4d60-b111-cc66514b84c9"
Oct 31 03:30:13.217114 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:13.217087    4447 scope.go:122] "RemoveContainer" containerID="32a2f90599d20871f953ea9a7ad91efebf07f3ffcbab9b2f158afe616a7494ce"
Oct 31 03:30:13.217501 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:30:13.217415    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890290-jz68t_odh-ai-gateway-infra(b24b7da8-1429-4d60-b111-cc66514b84c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t" podUID="b24b7da8-1429-4d60-b111-cc66514b84c9"
Oct 31 03:30:19.002665 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:30:19.002627    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:30:25.601145 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:25.601113    4447 scope.go:122] "RemoveContainer" containerID="32a2f90599d20871f953ea9a7ad91efebf07f3ffcbab9b2f158afe616a7494ce"
Oct 31 03:30:26.297017 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:26.296972    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t" event={"ID":"b24b7da8-1429-4d60-b111-cc66514b84c9","Type":"ContainerStarted","Data":"161c2cfdbe6033219a27ec4059fb58ce55cce043907a3dbca0a69eb9a1bb0102"}
Oct 31 03:30:26.526135 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:30:26.526104    4447 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24b7da8_1429_4d60_b111_cc66514b84c9.slice/crio-conmon-161c2cfdbe6033219a27ec4059fb58ce55cce043907a3dbca0a69eb9a1bb0102.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 03:30:27.305096 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:27.305064    4447 generic.go:356] "Generic (PLEG): container finished" podID="b24b7da8-1429-4d60-b111-cc66514b84c9" containerID="161c2cfdbe6033219a27ec4059fb58ce55cce043907a3dbca0a69eb9a1bb0102" exitCode=60
Oct 31 03:30:27.305576 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:27.305138    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t" event={"ID":"b24b7da8-1429-4d60-b111-cc66514b84c9","Type":"ContainerDied","Data":"161c2cfdbe6033219a27ec4059fb58ce55cce043907a3dbca0a69eb9a1bb0102"}
Oct 31 03:30:27.305576 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:27.305212    4447 scope.go:122] "RemoveContainer" containerID="32a2f90599d20871f953ea9a7ad91efebf07f3ffcbab9b2f158afe616a7494ce"
Oct 31 03:30:27.305671 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:27.305595    4447 scope.go:122] "RemoveContainer" containerID="161c2cfdbe6033219a27ec4059fb58ce55cce043907a3dbca0a69eb9a1bb0102"
Oct 31 03:30:27.305866 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:30:27.305842    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890290-jz68t_odh-ai-gateway-infra(b24b7da8-1429-4d60-b111-cc66514b84c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t" podUID="b24b7da8-1429-4d60-b111-cc66514b84c9"
Oct 31 03:30:27.434319 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:27.434262    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t"]
Oct 31 03:30:28.238743 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.238711    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v"]
Oct 31 03:30:28.244159 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.244142    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v"
Oct 31 03:30:28.297258 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.297227    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-b63425\""
Oct 31 03:30:28.297501 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.297395    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-b63425-dockercfg-5rbc4\""
Oct 31 03:30:28.301725 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.301699    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v"]
Oct 31 03:30:28.370233 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.370208    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq"]
Oct 31 03:30:28.374698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.374675    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq"
Oct 31 03:30:28.416663 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.416635    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh9b2\" (UniqueName: \"kubernetes.io/projected/09f019db-a086-4f4b-b1cf-499c6aacc158-kube-api-access-jh9b2\") pod \"payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v\" (UID: \"09f019db-a086-4f4b-b1cf-499c6aacc158\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v"
Oct 31 03:30:28.416819 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.416718    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f019db-a086-4f4b-b1cf-499c6aacc158-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v\" (UID: \"09f019db-a086-4f4b-b1cf-499c6aacc158\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v"
Oct 31 03:30:28.437017 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.436983    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq"]
Oct 31 03:30:28.480271 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.480249    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t"
Oct 31 03:30:28.519001 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.518924    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pmx2\" (UniqueName: \"kubernetes.io/projected/d02acebf-6463-454b-9c79-76c9cf0709fc-kube-api-access-7pmx2\") pod \"payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq\" (UID: \"d02acebf-6463-454b-9c79-76c9cf0709fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq"
Oct 31 03:30:28.519277 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.519232    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02acebf-6463-454b-9c79-76c9cf0709fc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq\" (UID: \"d02acebf-6463-454b-9c79-76c9cf0709fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq"
Oct 31 03:30:28.519516 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.519320    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh9b2\" (UniqueName: \"kubernetes.io/projected/09f019db-a086-4f4b-b1cf-499c6aacc158-kube-api-access-jh9b2\") pod \"payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v\" (UID: \"09f019db-a086-4f4b-b1cf-499c6aacc158\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v"
Oct 31 03:30:28.519764 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.519743    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f019db-a086-4f4b-b1cf-499c6aacc158-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v\" (UID: \"09f019db-a086-4f4b-b1cf-499c6aacc158\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v"
Oct 31 03:30:28.520428 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.520405    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f019db-a086-4f4b-b1cf-499c6aacc158-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v\" (UID: \"09f019db-a086-4f4b-b1cf-499c6aacc158\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v"
Oct 31 03:30:28.581212 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.581180    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh9b2\" (UniqueName: \"kubernetes.io/projected/09f019db-a086-4f4b-b1cf-499c6aacc158-kube-api-access-jh9b2\") pod \"payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v\" (UID: \"09f019db-a086-4f4b-b1cf-499c6aacc158\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v"
Oct 31 03:30:28.620261 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.620233    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24b7da8-1429-4d60-b111-cc66514b84c9-kube-api-access-q9r9g\" (UniqueName: \"kubernetes.io/projected/b24b7da8-1429-4d60-b111-cc66514b84c9-kube-api-access-q9r9g\") pod \"b24b7da8-1429-4d60-b111-cc66514b84c9\" (UID: \"b24b7da8-1429-4d60-b111-cc66514b84c9\") "
Oct 31 03:30:28.620407 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.620327    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b24b7da8-1429-4d60-b111-cc66514b84c9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b24b7da8-1429-4d60-b111-cc66514b84c9-maas-api-service-ca\") pod \"b24b7da8-1429-4d60-b111-cc66514b84c9\" (UID: \"b24b7da8-1429-4d60-b111-cc66514b84c9\") "
Oct 31 03:30:28.620544 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.620524    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pmx2\" (UniqueName: \"kubernetes.io/projected/d02acebf-6463-454b-9c79-76c9cf0709fc-kube-api-access-7pmx2\") pod \"payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq\" (UID: \"d02acebf-6463-454b-9c79-76c9cf0709fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq"
Oct 31 03:30:28.620627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.620610    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02acebf-6463-454b-9c79-76c9cf0709fc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq\" (UID: \"d02acebf-6463-454b-9c79-76c9cf0709fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq"
Oct 31 03:30:28.620690 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.620668    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24b7da8-1429-4d60-b111-cc66514b84c9-maas-api-service-ca" pod "b24b7da8-1429-4d60-b111-cc66514b84c9" (UID: "b24b7da8-1429-4d60-b111-cc66514b84c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:28.620852 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.620832    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b24b7da8-1429-4d60-b111-cc66514b84c9-maas-api-service-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:30:28.621663 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.621639    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02acebf-6463-454b-9c79-76c9cf0709fc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq\" (UID: \"d02acebf-6463-454b-9c79-76c9cf0709fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq"
Oct 31 03:30:28.622693 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.622676    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24b7da8-1429-4d60-b111-cc66514b84c9-kube-api-access-q9r9g" pod "b24b7da8-1429-4d60-b111-cc66514b84c9" (UID: "b24b7da8-1429-4d60-b111-cc66514b84c9"). InnerVolumeSpecName "kube-api-access-q9r9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:28.684551 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.684520    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pmx2\" (UniqueName: \"kubernetes.io/projected/d02acebf-6463-454b-9c79-76c9cf0709fc-kube-api-access-7pmx2\") pod \"payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq\" (UID: \"d02acebf-6463-454b-9c79-76c9cf0709fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq"
Oct 31 03:30:28.689395 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.689371    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq"
Oct 31 03:30:28.724701 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.721872    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9r9g\" (UniqueName: \"kubernetes.io/projected/b24b7da8-1429-4d60-b111-cc66514b84c9-kube-api-access-q9r9g\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:30:28.858712 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:28.858616    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v"
Oct 31 03:30:29.073159 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:29.073133    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq"]
Oct 31 03:30:29.073922 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:30:29.073854    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02acebf_6463_454b_9c79_76c9cf0709fc.slice/crio-fd6264734c1a7197b77b5f79b736d0539951ad94a4cce8275064b29e7b85ec61 WatchSource:0}: Error finding container fd6264734c1a7197b77b5f79b736d0539951ad94a4cce8275064b29e7b85ec61: Status 404 returned error can't find the container with id fd6264734c1a7197b77b5f79b736d0539951ad94a4cce8275064b29e7b85ec61
Oct 31 03:30:29.241635 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:29.241607    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v"]
Oct 31 03:30:29.242588 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:30:29.242558    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f019db_a086_4f4b_b1cf_499c6aacc158.slice/crio-765a36e820e01a9d89a780622dbf918636d9e614deab0bba6f191df915aa2b22 WatchSource:0}: Error finding container 765a36e820e01a9d89a780622dbf918636d9e614deab0bba6f191df915aa2b22: Status 404 returned error can't find the container with id 765a36e820e01a9d89a780622dbf918636d9e614deab0bba6f191df915aa2b22
Oct 31 03:30:29.341479 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:29.341437    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq" event={"ID":"d02acebf-6463-454b-9c79-76c9cf0709fc","Type":"ContainerStarted","Data":"75b0f17d630c7ea31df0a54af6f1273ddcef4638456587653073662e47603cea"}
Oct 31 03:30:29.341637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:29.341499    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq" event={"ID":"d02acebf-6463-454b-9c79-76c9cf0709fc","Type":"ContainerStarted","Data":"fd6264734c1a7197b77b5f79b736d0539951ad94a4cce8275064b29e7b85ec61"}
Oct 31 03:30:29.341637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:29.341521    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq"
Oct 31 03:30:29.343089 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:29.343058    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v" event={"ID":"09f019db-a086-4f4b-b1cf-499c6aacc158","Type":"ContainerStarted","Data":"efb1846a9f7efd209b7615399aa53308e58ad4605d88efaa119f0533d5c10a62"}
Oct 31 03:30:29.343214 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:29.343106    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v"
Oct 31 03:30:29.343214 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:29.343124    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v" event={"ID":"09f019db-a086-4f4b-b1cf-499c6aacc158","Type":"ContainerStarted","Data":"765a36e820e01a9d89a780622dbf918636d9e614deab0bba6f191df915aa2b22"}
Oct 31 03:30:29.344270 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:29.344244    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t" event={"ID":"b24b7da8-1429-4d60-b111-cc66514b84c9","Type":"ContainerDied","Data":"30b54183ba38c8b5a6be16a81d3e0aa31749b4611d0021870fa1f6e1382ca34d"}
Oct 31 03:30:29.344270 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:29.344260    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t"
Oct 31 03:30:29.344408 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:29.344296    4447 scope.go:122] "RemoveContainer" containerID="161c2cfdbe6033219a27ec4059fb58ce55cce043907a3dbca0a69eb9a1bb0102"
Oct 31 03:30:29.475413 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:29.475364    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq" podStartSLOduration=1.475350443 podStartE2EDuration="1.475350443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:29.472067299 +0000 UTC m=+654.983049715" watchObservedRunningTime="2026-10-31 03:30:29.475350443 +0000 UTC m=+654.986332819"
Oct 31 03:30:29.594907 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:29.593947    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t"]
Oct 31 03:30:29.597778 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:29.597750    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890290-jz68t"]
Oct 31 03:30:29.716282 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:29.716225    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v" podStartSLOduration=1.716211341 podStartE2EDuration="1.716211341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:29.714456183 +0000 UTC m=+655.225438564" watchObservedRunningTime="2026-10-31 03:30:29.716211341 +0000 UTC m=+655.227193717"
Oct 31 03:30:30.556197 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.556163    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"]
Oct 31 03:30:30.556721 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.556708    4447 cpu_manager.go:414] "containerMap: removing container" podUID="b24b7da8-1429-4d60-b111-cc66514b84c9" containerName="cleanup"
Oct 31 03:30:30.556767 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.556725    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24b7da8-1429-4d60-b111-cc66514b84c9" containerName="cleanup"
Oct 31 03:30:30.556767 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.556739    4447 cpu_manager.go:414] "containerMap: removing container" podUID="b24b7da8-1429-4d60-b111-cc66514b84c9" containerName="cleanup"
Oct 31 03:30:30.556767 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.556744    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24b7da8-1429-4d60-b111-cc66514b84c9" containerName="cleanup"
Oct 31 03:30:30.556854 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.556829    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24b7da8-1429-4d60-b111-cc66514b84c9" containerName="cleanup"
Oct 31 03:30:30.556854 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.556837    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24b7da8-1429-4d60-b111-cc66514b84c9" containerName="cleanup"
Oct 31 03:30:30.556854 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.556846    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24b7da8-1429-4d60-b111-cc66514b84c9" containerName="cleanup"
Oct 31 03:30:30.561467 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.561448    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"
Oct 31 03:30:30.607452 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.607426    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b24b7da8-1429-4d60-b111-cc66514b84c9" path="/var/lib/kubelet/pods/b24b7da8-1429-4d60-b111-cc66514b84c9/volumes"
Oct 31 03:30:30.642289 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.642253    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da528f3a-515b-4c82-b113-aed67ef952db-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2\" (UID: \"da528f3a-515b-4c82-b113-aed67ef952db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"
Oct 31 03:30:30.642464 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.642314    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da528f3a-515b-4c82-b113-aed67ef952db-metrics-certs\") pod \"maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2\" (UID: \"da528f3a-515b-4c82-b113-aed67ef952db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"
Oct 31 03:30:30.642464 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.642339    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da528f3a-515b-4c82-b113-aed67ef952db-maas-api-tls\") pod \"maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2\" (UID: \"da528f3a-515b-4c82-b113-aed67ef952db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"
Oct 31 03:30:30.642599 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.642547    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4669l\" (UniqueName: \"kubernetes.io/projected/da528f3a-515b-4c82-b113-aed67ef952db-kube-api-access-4669l\") pod \"maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2\" (UID: \"da528f3a-515b-4c82-b113-aed67ef952db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"
Oct 31 03:30:30.666419 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.666384    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-b63425-serving-cert\""
Oct 31 03:30:30.666419 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.666414    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-spbn9\""
Oct 31 03:30:30.666606 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.666428    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 03:30:30.669062 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.669045    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"]
Oct 31 03:30:30.743216 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.743186    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da528f3a-515b-4c82-b113-aed67ef952db-metrics-certs\") pod \"maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2\" (UID: \"da528f3a-515b-4c82-b113-aed67ef952db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"
Oct 31 03:30:30.743216 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.743220    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da528f3a-515b-4c82-b113-aed67ef952db-maas-api-tls\") pod \"maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2\" (UID: \"da528f3a-515b-4c82-b113-aed67ef952db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"
Oct 31 03:30:30.743440 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.743294    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4669l\" (UniqueName: \"kubernetes.io/projected/da528f3a-515b-4c82-b113-aed67ef952db-kube-api-access-4669l\") pod \"maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2\" (UID: \"da528f3a-515b-4c82-b113-aed67ef952db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"
Oct 31 03:30:30.743440 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:30:30.743396    4447 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-serving-cert: secret "maas-api-e2e-worker-w6-b63425-serving-cert" not found
Oct 31 03:30:30.743440 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.743431    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da528f3a-515b-4c82-b113-aed67ef952db-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2\" (UID: \"da528f3a-515b-4c82-b113-aed67ef952db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"
Oct 31 03:30:30.743590 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:30:30.743456    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da528f3a-515b-4c82-b113-aed67ef952db-maas-api-tls podName:da528f3a-515b-4c82-b113-aed67ef952db nodeName:}" failed. No retries permitted until 2026-10-31 03:30:31.243441093 +0000 UTC m=+656.754423461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da528f3a-515b-4c82-b113-aed67ef952db-maas-api-tls") pod "maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2" (UID: "da528f3a-515b-4c82-b113-aed67ef952db") : secret "maas-api-e2e-worker-w6-b63425-serving-cert" not found
Oct 31 03:30:30.744043 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.743981    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da528f3a-515b-4c82-b113-aed67ef952db-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2\" (UID: \"da528f3a-515b-4c82-b113-aed67ef952db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"
Oct 31 03:30:30.745748 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.745730    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da528f3a-515b-4c82-b113-aed67ef952db-metrics-certs\") pod \"maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2\" (UID: \"da528f3a-515b-4c82-b113-aed67ef952db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"
Oct 31 03:30:30.805231 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:30.805203    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4669l\" (UniqueName: \"kubernetes.io/projected/da528f3a-515b-4c82-b113-aed67ef952db-kube-api-access-4669l\") pod \"maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2\" (UID: \"da528f3a-515b-4c82-b113-aed67ef952db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"
Oct 31 03:30:31.247705 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:31.247661    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da528f3a-515b-4c82-b113-aed67ef952db-maas-api-tls\") pod \"maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2\" (UID: \"da528f3a-515b-4c82-b113-aed67ef952db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"
Oct 31 03:30:31.250442 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:31.250408    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da528f3a-515b-4c82-b113-aed67ef952db-maas-api-tls\") pod \"maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2\" (UID: \"da528f3a-515b-4c82-b113-aed67ef952db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"
Oct 31 03:30:31.476731 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:31.476696    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"
Oct 31 03:30:31.860971 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:31.859801    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"]
Oct 31 03:30:31.862926 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:30:31.862878    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda528f3a_515b_4c82_b113_aed67ef952db.slice/crio-5d5d8f7802147fb61c4e363fb67ff0d17f9b80580b4f42766eb7f4270c57ad89 WatchSource:0}: Error finding container 5d5d8f7802147fb61c4e363fb67ff0d17f9b80580b4f42766eb7f4270c57ad89: Status 404 returned error can't find the container with id 5d5d8f7802147fb61c4e363fb67ff0d17f9b80580b4f42766eb7f4270c57ad89
Oct 31 03:30:32.365296 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:32.365256    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2" event={"ID":"da528f3a-515b-4c82-b113-aed67ef952db","Type":"ContainerStarted","Data":"5d5d8f7802147fb61c4e363fb67ff0d17f9b80580b4f42766eb7f4270c57ad89"}
Oct 31 03:30:33.373875 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:33.373836    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2" event={"ID":"da528f3a-515b-4c82-b113-aed67ef952db","Type":"ContainerStarted","Data":"5a73cb7d4ccaf9c91670a59969cffec42fc99d83d0a6205e7122e64e03c53844"}
Oct 31 03:30:33.374348 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:33.374067    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"
Oct 31 03:30:33.495906 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:33.495820    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2" podStartSLOduration=2.611971915 podStartE2EDuration="3.495801143s" totalImagesPullingTime="883.829228ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:33.492025262 +0000 UTC m=+659.003007665" watchObservedRunningTime="2026-10-31 03:30:33.495801143 +0000 UTC m=+659.006783521"
Oct 31 03:30:35.480556 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:35.480518    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt"]
Oct 31 03:30:35.481098 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:35.481084    4447 cpu_manager.go:414] "containerMap: removing container" podUID="b24b7da8-1429-4d60-b111-cc66514b84c9" containerName="cleanup"
Oct 31 03:30:35.481158 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:35.481099    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24b7da8-1429-4d60-b111-cc66514b84c9" containerName="cleanup"
Oct 31 03:30:35.485793 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:35.485771    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt"
Oct 31 03:30:35.540706 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:35.540663    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-b98893\""
Oct 31 03:30:35.590681 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:35.590643    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-b98893-dockercfg-5m2nk\""
Oct 31 03:30:35.593998 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:35.593965    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72c4ce5c-8bd0-4857-97cc-8f26437cd57c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt\" (UID: \"72c4ce5c-8bd0-4857-97cc-8f26437cd57c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt"
Oct 31 03:30:35.594232 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:35.594201    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjcc8\" (UniqueName: \"kubernetes.io/projected/72c4ce5c-8bd0-4857-97cc-8f26437cd57c-kube-api-access-mjcc8\") pod \"payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt\" (UID: \"72c4ce5c-8bd0-4857-97cc-8f26437cd57c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt"
Oct 31 03:30:35.594488 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:35.594470    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt"]
Oct 31 03:30:35.597107 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:35.597085    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv"]
Oct 31 03:30:35.601196 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:35.601177    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv"
Oct 31 03:30:35.650496 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:35.650450    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv"]
Oct 31 03:30:35.695206 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:35.695175    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4tvv\" (UniqueName: \"kubernetes.io/projected/b07daa8e-894c-42e3-ac95-a21c71b8735c-kube-api-access-w4tvv\") pod \"payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv\" (UID: \"b07daa8e-894c-42e3-ac95-a21c71b8735c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv"
Oct 31 03:30:35.695365 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:35.695268    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjcc8\" (UniqueName: \"kubernetes.io/projected/72c4ce5c-8bd0-4857-97cc-8f26437cd57c-kube-api-access-mjcc8\") pod \"payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt\" (UID: \"72c4ce5c-8bd0-4857-97cc-8f26437cd57c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt"
Oct 31 03:30:35.695365 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:35.695309    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72c4ce5c-8bd0-4857-97cc-8f26437cd57c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt\" (UID: \"72c4ce5c-8bd0-4857-97cc-8f26437cd57c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt"
Oct 31 03:30:35.695365 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:35.695334    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07daa8e-894c-42e3-ac95-a21c71b8735c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv\" (UID: \"b07daa8e-894c-42e3-ac95-a21c71b8735c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv"
Oct 31 03:30:35.696025 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:35.696007    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72c4ce5c-8bd0-4857-97cc-8f26437cd57c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt\" (UID: \"72c4ce5c-8bd0-4857-97cc-8f26437cd57c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt"
Oct 31 03:30:35.757037 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:35.756967    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjcc8\" (UniqueName: \"kubernetes.io/projected/72c4ce5c-8bd0-4857-97cc-8f26437cd57c-kube-api-access-mjcc8\") pod \"payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt\" (UID: \"72c4ce5c-8bd0-4857-97cc-8f26437cd57c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt"
Oct 31 03:30:35.796153 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:35.796117    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07daa8e-894c-42e3-ac95-a21c71b8735c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv\" (UID: \"b07daa8e-894c-42e3-ac95-a21c71b8735c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv"
Oct 31 03:30:35.796316 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:35.796222    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4tvv\" (UniqueName: \"kubernetes.io/projected/b07daa8e-894c-42e3-ac95-a21c71b8735c-kube-api-access-w4tvv\") pod \"payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv\" (UID: \"b07daa8e-894c-42e3-ac95-a21c71b8735c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv"
Oct 31 03:30:35.796896 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:35.796866    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07daa8e-894c-42e3-ac95-a21c71b8735c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv\" (UID: \"b07daa8e-894c-42e3-ac95-a21c71b8735c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv"
Oct 31 03:30:35.808622 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:35.808594    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt"
Oct 31 03:30:35.860026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:35.858702    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4tvv\" (UniqueName: \"kubernetes.io/projected/b07daa8e-894c-42e3-ac95-a21c71b8735c-kube-api-access-w4tvv\") pod \"payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv\" (UID: \"b07daa8e-894c-42e3-ac95-a21c71b8735c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv"
Oct 31 03:30:35.916674 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:35.916645    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv"
Oct 31 03:30:36.184156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:36.184110    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt"]
Oct 31 03:30:36.185584 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:30:36.185556    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c4ce5c_8bd0_4857_97cc_8f26437cd57c.slice/crio-5e9e58a3ea630604bca7c2e7c2f13a3047f3618197ef0575589f471adb5ab669 WatchSource:0}: Error finding container 5e9e58a3ea630604bca7c2e7c2f13a3047f3618197ef0575589f471adb5ab669: Status 404 returned error can't find the container with id 5e9e58a3ea630604bca7c2e7c2f13a3047f3618197ef0575589f471adb5ab669
Oct 31 03:30:36.293594 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:36.293568    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv"]
Oct 31 03:30:36.294811 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:30:36.294783    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07daa8e_894c_42e3_ac95_a21c71b8735c.slice/crio-6fbc2b4e79c5f45734fc9280f1002fbadb9fb086e325c76666ef1bbf8fb5012c WatchSource:0}: Error finding container 6fbc2b4e79c5f45734fc9280f1002fbadb9fb086e325c76666ef1bbf8fb5012c: Status 404 returned error can't find the container with id 6fbc2b4e79c5f45734fc9280f1002fbadb9fb086e325c76666ef1bbf8fb5012c
Oct 31 03:30:36.393119 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:36.393083    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv" event={"ID":"b07daa8e-894c-42e3-ac95-a21c71b8735c","Type":"ContainerStarted","Data":"871763a11296cc4e181fa04591bc59c9c48414aae88c446ad30e0737edce9642"}
Oct 31 03:30:36.393369 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:36.393143    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv" event={"ID":"b07daa8e-894c-42e3-ac95-a21c71b8735c","Type":"ContainerStarted","Data":"6fbc2b4e79c5f45734fc9280f1002fbadb9fb086e325c76666ef1bbf8fb5012c"}
Oct 31 03:30:36.393369 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:36.393174    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv"
Oct 31 03:30:36.394805 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:36.394780    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt" event={"ID":"72c4ce5c-8bd0-4857-97cc-8f26437cd57c","Type":"ContainerStarted","Data":"1f094289fd45074247364f13a81b8d6609c8c30601e31576ddaeb5b4b5232433"}
Oct 31 03:30:36.394942 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:36.394819    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt" event={"ID":"72c4ce5c-8bd0-4857-97cc-8f26437cd57c","Type":"ContainerStarted","Data":"5e9e58a3ea630604bca7c2e7c2f13a3047f3618197ef0575589f471adb5ab669"}
Oct 31 03:30:36.394986 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:36.394936    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt"
Oct 31 03:30:36.511798 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:36.511740    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv" podStartSLOduration=1.5117262070000002 podStartE2EDuration="1.511726207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:36.509429994 +0000 UTC m=+662.020412368" watchObservedRunningTime="2026-10-31 03:30:36.511726207 +0000 UTC m=+662.022708583"
Oct 31 03:30:36.629118 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:36.629065    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt" podStartSLOduration=1.6290525009999999 podStartE2EDuration="1.629052501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:36.62705949 +0000 UTC m=+662.138041927" watchObservedRunningTime="2026-10-31 03:30:36.629052501 +0000 UTC m=+662.140034877"
Oct 31 03:30:38.056953 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:38.056915    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd"]
Oct 31 03:30:38.061114 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:38.061093    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd"
Oct 31 03:30:38.120083 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:38.120050    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd"]
Oct 31 03:30:38.123879 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:38.123847    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxkbl\" (UniqueName: \"kubernetes.io/projected/a48bb0ee-3ca9-48f2-94e6-9b9d201a531a-kube-api-access-fxkbl\") pod \"payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd\" (UID: \"a48bb0ee-3ca9-48f2-94e6-9b9d201a531a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd"
Oct 31 03:30:38.124007 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:38.123941    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48bb0ee-3ca9-48f2-94e6-9b9d201a531a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd\" (UID: \"a48bb0ee-3ca9-48f2-94e6-9b9d201a531a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd"
Oct 31 03:30:38.165673 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:38.165639    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-e00ad3\""
Oct 31 03:30:38.166286 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:38.166264    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-e00ad3-dockercfg-5hhrm\""
Oct 31 03:30:38.219226 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:38.219195    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s"]
Oct 31 03:30:38.223205 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:38.223190    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s"
Oct 31 03:30:38.225094 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:38.225077    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxkbl\" (UniqueName: \"kubernetes.io/projected/a48bb0ee-3ca9-48f2-94e6-9b9d201a531a-kube-api-access-fxkbl\") pod \"payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd\" (UID: \"a48bb0ee-3ca9-48f2-94e6-9b9d201a531a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd"
Oct 31 03:30:38.225153 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:38.225120    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48bb0ee-3ca9-48f2-94e6-9b9d201a531a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd\" (UID: \"a48bb0ee-3ca9-48f2-94e6-9b9d201a531a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd"
Oct 31 03:30:38.225805 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:38.225786    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48bb0ee-3ca9-48f2-94e6-9b9d201a531a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd\" (UID: \"a48bb0ee-3ca9-48f2-94e6-9b9d201a531a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd"
Oct 31 03:30:38.282081 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:38.282041    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s"]
Oct 31 03:30:38.287140 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:38.287112    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxkbl\" (UniqueName: \"kubernetes.io/projected/a48bb0ee-3ca9-48f2-94e6-9b9d201a531a-kube-api-access-fxkbl\") pod \"payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd\" (UID: \"a48bb0ee-3ca9-48f2-94e6-9b9d201a531a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd"
Oct 31 03:30:38.326635 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:38.326550    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4dd4\" (UniqueName: \"kubernetes.io/projected/21a7823a-b004-4b58-891e-bc463000308f-kube-api-access-w4dd4\") pod \"payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s\" (UID: \"21a7823a-b004-4b58-891e-bc463000308f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s"
Oct 31 03:30:38.326784 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:38.326736    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a7823a-b004-4b58-891e-bc463000308f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s\" (UID: \"21a7823a-b004-4b58-891e-bc463000308f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s"
Oct 31 03:30:38.374944 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:38.374866    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd"
Oct 31 03:30:38.429451 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:38.428554    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4dd4\" (UniqueName: \"kubernetes.io/projected/21a7823a-b004-4b58-891e-bc463000308f-kube-api-access-w4dd4\") pod \"payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s\" (UID: \"21a7823a-b004-4b58-891e-bc463000308f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s"
Oct 31 03:30:38.429451 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:38.428764    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a7823a-b004-4b58-891e-bc463000308f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s\" (UID: \"21a7823a-b004-4b58-891e-bc463000308f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s"
Oct 31 03:30:38.429721 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:38.429621    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a7823a-b004-4b58-891e-bc463000308f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s\" (UID: \"21a7823a-b004-4b58-891e-bc463000308f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s"
Oct 31 03:30:38.490419 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:38.490393    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4dd4\" (UniqueName: \"kubernetes.io/projected/21a7823a-b004-4b58-891e-bc463000308f-kube-api-access-w4dd4\") pod \"payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s\" (UID: \"21a7823a-b004-4b58-891e-bc463000308f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s"
Oct 31 03:30:38.537351 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:38.537322    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s"
Oct 31 03:30:38.752875 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:38.752843    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd"]
Oct 31 03:30:38.753011 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:30:38.752940    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48bb0ee_3ca9_48f2_94e6_9b9d201a531a.slice/crio-9c705dd6d5ded6d56fec93b9bb992deb66cbbfc9d1a44efca42c40f9bd0a74a5 WatchSource:0}: Error finding container 9c705dd6d5ded6d56fec93b9bb992deb66cbbfc9d1a44efca42c40f9bd0a74a5: Status 404 returned error can't find the container with id 9c705dd6d5ded6d56fec93b9bb992deb66cbbfc9d1a44efca42c40f9bd0a74a5
Oct 31 03:30:38.915811 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:38.915784    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s"]
Oct 31 03:30:38.916098 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:30:38.916077    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a7823a_b004_4b58_891e_bc463000308f.slice/crio-67918b335dbf391b420e868ef5758ced253c1aa195c877576f2288ce11fdf17a WatchSource:0}: Error finding container 67918b335dbf391b420e868ef5758ced253c1aa195c877576f2288ce11fdf17a: Status 404 returned error can't find the container with id 67918b335dbf391b420e868ef5758ced253c1aa195c877576f2288ce11fdf17a
Oct 31 03:30:39.386079 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:39.386049    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"
Oct 31 03:30:39.415216 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:39.415183    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd" event={"ID":"a48bb0ee-3ca9-48f2-94e6-9b9d201a531a","Type":"ContainerStarted","Data":"6a5cd286fad0a1ad11f3f99660b6de1d383df65540230e139cef173c2702b9c7"}
Oct 31 03:30:39.415402 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:39.415238    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd"
Oct 31 03:30:39.415402 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:39.415259    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd" event={"ID":"a48bb0ee-3ca9-48f2-94e6-9b9d201a531a","Type":"ContainerStarted","Data":"9c705dd6d5ded6d56fec93b9bb992deb66cbbfc9d1a44efca42c40f9bd0a74a5"}
Oct 31 03:30:39.416795 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:39.416763    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s" event={"ID":"21a7823a-b004-4b58-891e-bc463000308f","Type":"ContainerStarted","Data":"d953e282332e56b84f9f7e676e0585cf49d623a30d31f2318dbfb05df24f7b2f"}
Oct 31 03:30:39.416950 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:39.416807    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s" event={"ID":"21a7823a-b004-4b58-891e-bc463000308f","Type":"ContainerStarted","Data":"67918b335dbf391b420e868ef5758ced253c1aa195c877576f2288ce11fdf17a"}
Oct 31 03:30:39.416950 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:39.416834    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s"
Oct 31 03:30:39.738690 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:39.738641    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s" podStartSLOduration=1.738625563 podStartE2EDuration="1.738625563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:39.622096807 +0000 UTC m=+665.133079183" watchObservedRunningTime="2026-10-31 03:30:39.738625563 +0000 UTC m=+665.249607941"
Oct 31 03:30:39.738970 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:39.738916    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd" podStartSLOduration=1.738910803 podStartE2EDuration="1.738910803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:39.737655125 +0000 UTC m=+665.248637501" watchObservedRunningTime="2026-10-31 03:30:39.738910803 +0000 UTC m=+665.249893225"
Oct 31 03:30:40.354651 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:40.354624    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq"
Oct 31 03:30:40.354820 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:40.354688    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v"
Oct 31 03:30:41.238122 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:41.238087    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"]
Oct 31 03:30:41.242333 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:41.242314    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"
Oct 31 03:30:41.346979 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:41.346953    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-e00ad3-serving-cert\""
Oct 31 03:30:41.349959 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:41.349938    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"]
Oct 31 03:30:41.362030 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:41.362007    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh\" (UID: \"502e9c23-5ef9-4da4-93c4-5a5755d1f61e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"
Oct 31 03:30:41.362143 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:41.362046    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh\" (UID: \"502e9c23-5ef9-4da4-93c4-5a5755d1f61e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"
Oct 31 03:30:41.362143 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:41.362062    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fpxj\" (UniqueName: \"kubernetes.io/projected/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-kube-api-access-4fpxj\") pod \"maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh\" (UID: \"502e9c23-5ef9-4da4-93c4-5a5755d1f61e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"
Oct 31 03:30:41.362220 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:41.362177    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-metrics-certs\") pod \"maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh\" (UID: \"502e9c23-5ef9-4da4-93c4-5a5755d1f61e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"
Oct 31 03:30:41.463461 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:41.463432    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh\" (UID: \"502e9c23-5ef9-4da4-93c4-5a5755d1f61e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"
Oct 31 03:30:41.463461 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:41.463465    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fpxj\" (UniqueName: \"kubernetes.io/projected/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-kube-api-access-4fpxj\") pod \"maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh\" (UID: \"502e9c23-5ef9-4da4-93c4-5a5755d1f61e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"
Oct 31 03:30:41.463693 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:41.463504    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-metrics-certs\") pod \"maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh\" (UID: \"502e9c23-5ef9-4da4-93c4-5a5755d1f61e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"
Oct 31 03:30:41.463693 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:30:41.463597    4447 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-serving-cert: secret "maas-api-e2e-worker-w1-e00ad3-serving-cert" not found
Oct 31 03:30:41.463693 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:41.463645    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh\" (UID: \"502e9c23-5ef9-4da4-93c4-5a5755d1f61e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"
Oct 31 03:30:41.463693 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:30:41.463670    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-maas-api-tls podName:502e9c23-5ef9-4da4-93c4-5a5755d1f61e nodeName:}" failed. No retries permitted until 2026-10-31 03:30:41.963641482 +0000 UTC m=+667.474623840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-maas-api-tls") pod "maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh" (UID: "502e9c23-5ef9-4da4-93c4-5a5755d1f61e") : secret "maas-api-e2e-worker-w1-e00ad3-serving-cert" not found
Oct 31 03:30:41.464358 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:41.464341    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh\" (UID: \"502e9c23-5ef9-4da4-93c4-5a5755d1f61e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"
Oct 31 03:30:41.466199 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:41.466182    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-metrics-certs\") pod \"maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh\" (UID: \"502e9c23-5ef9-4da4-93c4-5a5755d1f61e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"
Oct 31 03:30:41.527270 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:41.527199    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fpxj\" (UniqueName: \"kubernetes.io/projected/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-kube-api-access-4fpxj\") pod \"maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh\" (UID: \"502e9c23-5ef9-4da4-93c4-5a5755d1f61e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"
Oct 31 03:30:41.969021 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:41.968990    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh\" (UID: \"502e9c23-5ef9-4da4-93c4-5a5755d1f61e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"
Oct 31 03:30:41.971567 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:41.971543    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh\" (UID: \"502e9c23-5ef9-4da4-93c4-5a5755d1f61e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"
Oct 31 03:30:42.156081 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:42.156050    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"
Oct 31 03:30:42.533629 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:42.533603    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"]
Oct 31 03:30:42.534740 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:30:42.534712    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod502e9c23_5ef9_4da4_93c4_5a5755d1f61e.slice/crio-819b848198a7e82b4c29ea11288b75cede5dbbf5c72b02c4c311d041d9d1ddf8 WatchSource:0}: Error finding container 819b848198a7e82b4c29ea11288b75cede5dbbf5c72b02c4c311d041d9d1ddf8: Status 404 returned error can't find the container with id 819b848198a7e82b4c29ea11288b75cede5dbbf5c72b02c4c311d041d9d1ddf8
Oct 31 03:30:42.883995 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:42.883916    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv"]
Oct 31 03:30:42.889177 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:42.889161    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv"
Oct 31 03:30:42.978855 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:42.978804    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6d58e8e-588d-43e8-808a-d672b68c033e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv\" (UID: \"e6d58e8e-588d-43e8-808a-d672b68c033e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv"
Oct 31 03:30:42.979077 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:42.978996    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs6px\" (UniqueName: \"kubernetes.io/projected/e6d58e8e-588d-43e8-808a-d672b68c033e-kube-api-access-qs6px\") pod \"payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv\" (UID: \"e6d58e8e-588d-43e8-808a-d672b68c033e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv"
Oct 31 03:30:42.992580 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:42.992550    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-096812\""
Oct 31 03:30:42.992580 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:42.992556    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-096812-dockercfg-njx7g\""
Oct 31 03:30:42.995656 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:42.995631    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv"]
Oct 31 03:30:43.026208 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:43.026175    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb"]
Oct 31 03:30:43.030089 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:43.030070    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb"
Oct 31 03:30:43.080348 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:43.080318    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9mpk\" (UniqueName: \"kubernetes.io/projected/bec36425-b986-4ea2-9daa-c21328d0e8df-kube-api-access-h9mpk\") pod \"payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb\" (UID: \"bec36425-b986-4ea2-9daa-c21328d0e8df\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb"
Oct 31 03:30:43.080541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:43.080386    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6d58e8e-588d-43e8-808a-d672b68c033e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv\" (UID: \"e6d58e8e-588d-43e8-808a-d672b68c033e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv"
Oct 31 03:30:43.080541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:43.080518    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bec36425-b986-4ea2-9daa-c21328d0e8df-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb\" (UID: \"bec36425-b986-4ea2-9daa-c21328d0e8df\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb"
Oct 31 03:30:43.080643 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:43.080566    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs6px\" (UniqueName: \"kubernetes.io/projected/e6d58e8e-588d-43e8-808a-d672b68c033e-kube-api-access-qs6px\") pod \"payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv\" (UID: \"e6d58e8e-588d-43e8-808a-d672b68c033e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv"
Oct 31 03:30:43.081046 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:43.081030    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6d58e8e-588d-43e8-808a-d672b68c033e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv\" (UID: \"e6d58e8e-588d-43e8-808a-d672b68c033e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv"
Oct 31 03:30:43.088152 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:43.088119    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb"]
Oct 31 03:30:43.142944 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:43.142845    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs6px\" (UniqueName: \"kubernetes.io/projected/e6d58e8e-588d-43e8-808a-d672b68c033e-kube-api-access-qs6px\") pod \"payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv\" (UID: \"e6d58e8e-588d-43e8-808a-d672b68c033e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv"
Oct 31 03:30:43.181835 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:43.181804    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bec36425-b986-4ea2-9daa-c21328d0e8df-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb\" (UID: \"bec36425-b986-4ea2-9daa-c21328d0e8df\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb"
Oct 31 03:30:43.181985 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:43.181914    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9mpk\" (UniqueName: \"kubernetes.io/projected/bec36425-b986-4ea2-9daa-c21328d0e8df-kube-api-access-h9mpk\") pod \"payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb\" (UID: \"bec36425-b986-4ea2-9daa-c21328d0e8df\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb"
Oct 31 03:30:43.182493 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:43.182475    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bec36425-b986-4ea2-9daa-c21328d0e8df-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb\" (UID: \"bec36425-b986-4ea2-9daa-c21328d0e8df\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb"
Oct 31 03:30:43.203657 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:43.203629    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv"
Oct 31 03:30:43.243304 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:43.243275    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9mpk\" (UniqueName: \"kubernetes.io/projected/bec36425-b986-4ea2-9daa-c21328d0e8df-kube-api-access-h9mpk\") pod \"payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb\" (UID: \"bec36425-b986-4ea2-9daa-c21328d0e8df\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb"
Oct 31 03:30:43.343377 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:43.343350    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb"
Oct 31 03:30:43.442637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:43.442605    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh" event={"ID":"502e9c23-5ef9-4da4-93c4-5a5755d1f61e","Type":"ContainerStarted","Data":"2cecbc557288ecc00304c139afdf1b5ef662092258bb7902b532a531db883192"}
Oct 31 03:30:43.442792 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:43.442658    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh" event={"ID":"502e9c23-5ef9-4da4-93c4-5a5755d1f61e","Type":"ContainerStarted","Data":"819b848198a7e82b4c29ea11288b75cede5dbbf5c72b02c4c311d041d9d1ddf8"}
Oct 31 03:30:43.442792 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:43.442680    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"
Oct 31 03:30:43.564631 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:43.564581    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh" podStartSLOduration=1.803905491 podStartE2EDuration="2.56456672s" totalImagesPullingTime="760.661229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:43.562197934 +0000 UTC m=+669.073180313" watchObservedRunningTime="2026-10-31 03:30:43.56456672 +0000 UTC m=+669.075549098"
Oct 31 03:30:43.579782 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:43.579742    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv"]
Oct 31 03:30:43.617543 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:30:43.617498    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d58e8e_588d_43e8_808a_d672b68c033e.slice/crio-c3650c382f1af2f3e31fbca20bb85b4e7e27319ea665b036d9fafebdb9d28805 WatchSource:0}: Error finding container c3650c382f1af2f3e31fbca20bb85b4e7e27319ea665b036d9fafebdb9d28805: Status 404 returned error can't find the container with id c3650c382f1af2f3e31fbca20bb85b4e7e27319ea665b036d9fafebdb9d28805
Oct 31 03:30:43.721677 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:43.721649    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb"]
Oct 31 03:30:43.722366 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:30:43.722343    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec36425_b986_4ea2_9daa_c21328d0e8df.slice/crio-d35c5953df8ede80c7cb2087094f7feb1268d05001e8c28dc2c042ae85bd5697 WatchSource:0}: Error finding container d35c5953df8ede80c7cb2087094f7feb1268d05001e8c28dc2c042ae85bd5697: Status 404 returned error can't find the container with id d35c5953df8ede80c7cb2087094f7feb1268d05001e8c28dc2c042ae85bd5697
Oct 31 03:30:44.450427 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:44.450388    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb" event={"ID":"bec36425-b986-4ea2-9daa-c21328d0e8df","Type":"ContainerStarted","Data":"c52ca1ea78ccdc54023cf97a5500ca3d9a4e9b51ddb13dff18b3b2c7163d48d4"}
Oct 31 03:30:44.450653 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:44.450487    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb" event={"ID":"bec36425-b986-4ea2-9daa-c21328d0e8df","Type":"ContainerStarted","Data":"d35c5953df8ede80c7cb2087094f7feb1268d05001e8c28dc2c042ae85bd5697"}
Oct 31 03:30:44.450653 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:44.450517    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb"
Oct 31 03:30:44.452079 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:44.452051    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv" event={"ID":"e6d58e8e-588d-43e8-808a-d672b68c033e","Type":"ContainerStarted","Data":"2a1cb53180646242eecc7cf34606edb05cff26e1ab82d79c6c2d156ea49dd904"}
Oct 31 03:30:44.452192 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:44.452099    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv" event={"ID":"e6d58e8e-588d-43e8-808a-d672b68c033e","Type":"ContainerStarted","Data":"c3650c382f1af2f3e31fbca20bb85b4e7e27319ea665b036d9fafebdb9d28805"}
Oct 31 03:30:44.570042 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:44.569980    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb" podStartSLOduration=2.5699613059999997 podStartE2EDuration="2.569961306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:44.566696882 +0000 UTC m=+670.077679259" watchObservedRunningTime="2026-10-31 03:30:44.569961306 +0000 UTC m=+670.080943688"
Oct 31 03:30:44.684630 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:44.684578    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv" podStartSLOduration=2.68452307 podStartE2EDuration="2.68452307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:44.68202552 +0000 UTC m=+670.193007919" watchObservedRunningTime="2026-10-31 03:30:44.68452307 +0000 UTC m=+670.195505449"
Oct 31 03:30:45.458198 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:45.458168    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv"
Oct 31 03:30:47.404521 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:47.404439    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv"
Oct 31 03:30:47.404521 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:47.404500    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt"
Oct 31 03:30:47.758391 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:47.758353    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf"]
Oct 31 03:30:47.765371 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:47.765072    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf"
Oct 31 03:30:47.819222 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:47.819111    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-14cb5a-dockercfg-snl9r\""
Oct 31 03:30:47.819222 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:47.819140    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-14cb5a\""
Oct 31 03:30:47.822840 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:47.822812    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf"]
Oct 31 03:30:47.833841 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:47.833818    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade58b85-8ae0-4b92-b0f6-aef25f34878b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf\" (UID: \"ade58b85-8ae0-4b92-b0f6-aef25f34878b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf"
Oct 31 03:30:47.834019 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:47.833878    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26hp2\" (UniqueName: \"kubernetes.io/projected/ade58b85-8ae0-4b92-b0f6-aef25f34878b-kube-api-access-26hp2\") pod \"payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf\" (UID: \"ade58b85-8ae0-4b92-b0f6-aef25f34878b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf"
Oct 31 03:30:47.935679 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:47.935643    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade58b85-8ae0-4b92-b0f6-aef25f34878b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf\" (UID: \"ade58b85-8ae0-4b92-b0f6-aef25f34878b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf"
Oct 31 03:30:47.935850 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:47.935713    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26hp2\" (UniqueName: \"kubernetes.io/projected/ade58b85-8ae0-4b92-b0f6-aef25f34878b-kube-api-access-26hp2\") pod \"payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf\" (UID: \"ade58b85-8ae0-4b92-b0f6-aef25f34878b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf"
Oct 31 03:30:47.936606 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:47.936581    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade58b85-8ae0-4b92-b0f6-aef25f34878b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf\" (UID: \"ade58b85-8ae0-4b92-b0f6-aef25f34878b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf"
Oct 31 03:30:47.997162 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:47.997136    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26hp2\" (UniqueName: \"kubernetes.io/projected/ade58b85-8ae0-4b92-b0f6-aef25f34878b-kube-api-access-26hp2\") pod \"payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf\" (UID: \"ade58b85-8ae0-4b92-b0f6-aef25f34878b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf"
Oct 31 03:30:48.079704 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:48.079614    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf"
Oct 31 03:30:48.455303 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:48.455274    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf"]
Oct 31 03:30:48.455801 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:30:48.455765    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade58b85_8ae0_4b92_b0f6_aef25f34878b.slice/crio-e8e65000c9649ff4942075447a24995e581e33e7a2f6664fbf41e3fc106245b6 WatchSource:0}: Error finding container e8e65000c9649ff4942075447a24995e581e33e7a2f6664fbf41e3fc106245b6: Status 404 returned error can't find the container with id e8e65000c9649ff4942075447a24995e581e33e7a2f6664fbf41e3fc106245b6
Oct 31 03:30:48.481122 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:48.481092    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf" event={"ID":"ade58b85-8ae0-4b92-b0f6-aef25f34878b","Type":"ContainerStarted","Data":"e8e65000c9649ff4942075447a24995e581e33e7a2f6664fbf41e3fc106245b6"}
Oct 31 03:30:49.003603 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:30:49.003559    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:30:49.457014 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:49.456993    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"
Oct 31 03:30:49.488711 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:49.488675    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf" event={"ID":"ade58b85-8ae0-4b92-b0f6-aef25f34878b","Type":"ContainerStarted","Data":"e31493a0fc40f9c9ca17c58770766d0a9d53a1fbf154bf94591444b245f04eaf"}
Oct 31 03:30:49.488876 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:49.488742    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf"
Oct 31 03:30:49.697450 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:49.697403    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf" podStartSLOduration=2.697388363 podStartE2EDuration="2.697388363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:49.694290988 +0000 UTC m=+675.205273364" watchObservedRunningTime="2026-10-31 03:30:49.697388363 +0000 UTC m=+675.208370739"
Oct 31 03:30:50.427298 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:50.427264    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd"
Oct 31 03:30:50.427490 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:50.427343    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s"
Oct 31 03:30:52.682337 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:52.682307    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt"]
Oct 31 03:30:52.686445 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:52.686428    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt"
Oct 31 03:30:52.744433 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:52.744401    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt"]
Oct 31 03:30:52.790030 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:52.789999    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-dec4a0\""
Oct 31 03:30:52.790192 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:52.790035    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-dec4a0-dockercfg-rzffp\""
Oct 31 03:30:52.790444 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:52.790424    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmn47\" (UniqueName: \"kubernetes.io/projected/22c3bb42-e607-495f-942f-15072c8a0cf6-kube-api-access-nmn47\") pod \"payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt\" (UID: \"22c3bb42-e607-495f-942f-15072c8a0cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt"
Oct 31 03:30:52.790486 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:52.790468    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c3bb42-e607-495f-942f-15072c8a0cf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt\" (UID: \"22c3bb42-e607-495f-942f-15072c8a0cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt"
Oct 31 03:30:52.832865 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:52.832832    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp"]
Oct 31 03:30:52.836858 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:52.836841    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp"
Oct 31 03:30:52.892054 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:52.892019    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c3bb42-e607-495f-942f-15072c8a0cf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt\" (UID: \"22c3bb42-e607-495f-942f-15072c8a0cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt"
Oct 31 03:30:52.892587 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:52.892557    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmn47\" (UniqueName: \"kubernetes.io/projected/22c3bb42-e607-495f-942f-15072c8a0cf6-kube-api-access-nmn47\") pod \"payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt\" (UID: \"22c3bb42-e607-495f-942f-15072c8a0cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt"
Oct 31 03:30:52.892969 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:52.892948    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c3bb42-e607-495f-942f-15072c8a0cf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt\" (UID: \"22c3bb42-e607-495f-942f-15072c8a0cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt"
Oct 31 03:30:52.896030 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:52.896006    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp"]
Oct 31 03:30:52.954110 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:52.954080    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmn47\" (UniqueName: \"kubernetes.io/projected/22c3bb42-e607-495f-942f-15072c8a0cf6-kube-api-access-nmn47\") pod \"payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt\" (UID: \"22c3bb42-e607-495f-942f-15072c8a0cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt"
Oct 31 03:30:52.993562 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:52.993513    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67nns\" (UniqueName: \"kubernetes.io/projected/b6d42eac-1761-4ff4-a92c-9de57c7fb84a-kube-api-access-67nns\") pod \"payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp\" (UID: \"b6d42eac-1761-4ff4-a92c-9de57c7fb84a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp"
Oct 31 03:30:52.993742 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:52.993600    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d42eac-1761-4ff4-a92c-9de57c7fb84a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp\" (UID: \"b6d42eac-1761-4ff4-a92c-9de57c7fb84a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp"
Oct 31 03:30:53.000326 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:53.000304    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt"
Oct 31 03:30:53.094855 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:53.094819    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67nns\" (UniqueName: \"kubernetes.io/projected/b6d42eac-1761-4ff4-a92c-9de57c7fb84a-kube-api-access-67nns\") pod \"payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp\" (UID: \"b6d42eac-1761-4ff4-a92c-9de57c7fb84a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp"
Oct 31 03:30:53.095046 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:53.094903    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d42eac-1761-4ff4-a92c-9de57c7fb84a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp\" (UID: \"b6d42eac-1761-4ff4-a92c-9de57c7fb84a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp"
Oct 31 03:30:53.095592 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:53.095570    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d42eac-1761-4ff4-a92c-9de57c7fb84a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp\" (UID: \"b6d42eac-1761-4ff4-a92c-9de57c7fb84a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp"
Oct 31 03:30:53.157216 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:53.157185    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67nns\" (UniqueName: \"kubernetes.io/projected/b6d42eac-1761-4ff4-a92c-9de57c7fb84a-kube-api-access-67nns\") pod \"payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp\" (UID: \"b6d42eac-1761-4ff4-a92c-9de57c7fb84a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp"
Oct 31 03:30:53.379191 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:53.379158    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt"]
Oct 31 03:30:53.379989 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:30:53.379959    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c3bb42_e607_495f_942f_15072c8a0cf6.slice/crio-387cf4fe7f925dd4f649be0e931f3de0df016bbb6afa0374cb7e9ee8b182e04e WatchSource:0}: Error finding container 387cf4fe7f925dd4f649be0e931f3de0df016bbb6afa0374cb7e9ee8b182e04e: Status 404 returned error can't find the container with id 387cf4fe7f925dd4f649be0e931f3de0df016bbb6afa0374cb7e9ee8b182e04e
Oct 31 03:30:53.451283 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:53.451260    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp"
Oct 31 03:30:53.520344 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:53.520316    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt" event={"ID":"22c3bb42-e607-495f-942f-15072c8a0cf6","Type":"ContainerStarted","Data":"072c8bc8d8671e6930444bb66cb620e0ca4d3c38b8b9be366fc6962df2268eed"}
Oct 31 03:30:53.520495 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:53.520381    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt"
Oct 31 03:30:53.520495 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:53.520404    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt" event={"ID":"22c3bb42-e607-495f-942f-15072c8a0cf6","Type":"ContainerStarted","Data":"387cf4fe7f925dd4f649be0e931f3de0df016bbb6afa0374cb7e9ee8b182e04e"}
Oct 31 03:30:53.639383 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:53.639281    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt" podStartSLOduration=1.6392665499999999 podStartE2EDuration="1.63926655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:53.636712125 +0000 UTC m=+679.147694504" watchObservedRunningTime="2026-10-31 03:30:53.63926655 +0000 UTC m=+679.150248926"
Oct 31 03:30:53.827657 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:53.827631    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp"]
Oct 31 03:30:53.828178 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:30:53.828154    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d42eac_1761_4ff4_a92c_9de57c7fb84a.slice/crio-68c20c5da2f730f5de0a75e24d4de48cb45ef4af28c145a36cfa8da09fa71eef WatchSource:0}: Error finding container 68c20c5da2f730f5de0a75e24d4de48cb45ef4af28c145a36cfa8da09fa71eef: Status 404 returned error can't find the container with id 68c20c5da2f730f5de0a75e24d4de48cb45ef4af28c145a36cfa8da09fa71eef
Oct 31 03:30:54.528473 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:54.528436    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp" event={"ID":"b6d42eac-1761-4ff4-a92c-9de57c7fb84a","Type":"ContainerStarted","Data":"113abeae90c9a322ae0507db8d9255c748f61c3a61a0f66ae09511f641eb27f1"}
Oct 31 03:30:54.528634 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:54.528496    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp" event={"ID":"b6d42eac-1761-4ff4-a92c-9de57c7fb84a","Type":"ContainerStarted","Data":"68c20c5da2f730f5de0a75e24d4de48cb45ef4af28c145a36cfa8da09fa71eef"}
Oct 31 03:30:54.528634 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:54.528527    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp"
Oct 31 03:30:54.647223 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:54.647167    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp" podStartSLOduration=2.647150633 podStartE2EDuration="2.647150633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:54.644759952 +0000 UTC m=+680.155742330" watchObservedRunningTime="2026-10-31 03:30:54.647150633 +0000 UTC m=+680.158133009"
Oct 31 03:30:55.385339 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:55.385304    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk"]
Oct 31 03:30:55.389361 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:55.389334    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk"
Oct 31 03:30:55.449016 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:55.448991    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk"]
Oct 31 03:30:55.461770 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:55.461741    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb"
Oct 31 03:30:55.494344 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:55.494311    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-909365-dockercfg-nzxt2\""
Oct 31 03:30:55.494344 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:55.494311    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-909365\""
Oct 31 03:30:55.524493 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:55.524466    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de5b552a-3495-40e0-8f75-8fd08752474e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk\" (UID: \"de5b552a-3495-40e0-8f75-8fd08752474e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk"
Oct 31 03:30:55.524641 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:55.524505    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l95fq\" (UniqueName: \"kubernetes.io/projected/de5b552a-3495-40e0-8f75-8fd08752474e-kube-api-access-l95fq\") pod \"payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk\" (UID: \"de5b552a-3495-40e0-8f75-8fd08752474e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk"
Oct 31 03:30:55.625513 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:55.625473    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de5b552a-3495-40e0-8f75-8fd08752474e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk\" (UID: \"de5b552a-3495-40e0-8f75-8fd08752474e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk"
Oct 31 03:30:55.625712 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:55.625522    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l95fq\" (UniqueName: \"kubernetes.io/projected/de5b552a-3495-40e0-8f75-8fd08752474e-kube-api-access-l95fq\") pod \"payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk\" (UID: \"de5b552a-3495-40e0-8f75-8fd08752474e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk"
Oct 31 03:30:55.626467 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:55.626441    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de5b552a-3495-40e0-8f75-8fd08752474e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk\" (UID: \"de5b552a-3495-40e0-8f75-8fd08752474e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk"
Oct 31 03:30:55.687789 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:55.687703    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l95fq\" (UniqueName: \"kubernetes.io/projected/de5b552a-3495-40e0-8f75-8fd08752474e-kube-api-access-l95fq\") pod \"payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk\" (UID: \"de5b552a-3495-40e0-8f75-8fd08752474e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk"
Oct 31 03:30:55.703966 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:55.703939    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk"
Oct 31 03:30:56.085441 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:56.085411    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk"]
Oct 31 03:30:56.086545 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:30:56.086525    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5b552a_3495_40e0_8f75_8fd08752474e.slice/crio-02a69a8145d10f9777f88018b8a0cdf9e4374fc53317214d96e7bdf9f07098bf WatchSource:0}: Error finding container 02a69a8145d10f9777f88018b8a0cdf9e4374fc53317214d96e7bdf9f07098bf: Status 404 returned error can't find the container with id 02a69a8145d10f9777f88018b8a0cdf9e4374fc53317214d96e7bdf9f07098bf
Oct 31 03:30:56.468539 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:56.468509    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv"
Oct 31 03:30:56.544935 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:56.544872    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk" event={"ID":"de5b552a-3495-40e0-8f75-8fd08752474e","Type":"ContainerStarted","Data":"dbfec1b33592296f28be1ee9efa702406eeea9560bbd2ddd44930425ebf498bc"}
Oct 31 03:30:56.545124 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:56.544951    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk" event={"ID":"de5b552a-3495-40e0-8f75-8fd08752474e","Type":"ContainerStarted","Data":"02a69a8145d10f9777f88018b8a0cdf9e4374fc53317214d96e7bdf9f07098bf"}
Oct 31 03:30:56.545124 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:56.544973    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk"
Oct 31 03:30:56.702250 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:30:56.702200    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk" podStartSLOduration=1.70218312 podStartE2EDuration="1.70218312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:56.700062729 +0000 UTC m=+682.211045107" watchObservedRunningTime="2026-10-31 03:30:56.70218312 +0000 UTC m=+682.213165496"
Oct 31 03:31:00.499646 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:00.499614    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf"
Oct 31 03:31:02.061526 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:02.061488    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"]
Oct 31 03:31:02.065962 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:02.065937    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"
Oct 31 03:31:02.100381 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:02.100347    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbf932c9-3168-4564-aab2-5391dd41cc1a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-096812-856c94495f-zh5m8\" (UID: \"bbf932c9-3168-4564-aab2-5391dd41cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"
Oct 31 03:31:02.100381 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:02.100378    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf932c9-3168-4564-aab2-5391dd41cc1a-metrics-certs\") pod \"maas-api-e2e-worker-w4-096812-856c94495f-zh5m8\" (UID: \"bbf932c9-3168-4564-aab2-5391dd41cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"
Oct 31 03:31:02.100576 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:02.100419    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbf932c9-3168-4564-aab2-5391dd41cc1a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-096812-856c94495f-zh5m8\" (UID: \"bbf932c9-3168-4564-aab2-5391dd41cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"
Oct 31 03:31:02.100576 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:02.100517    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqrt6\" (UniqueName: \"kubernetes.io/projected/bbf932c9-3168-4564-aab2-5391dd41cc1a-kube-api-access-mqrt6\") pod \"maas-api-e2e-worker-w4-096812-856c94495f-zh5m8\" (UID: \"bbf932c9-3168-4564-aab2-5391dd41cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"
Oct 31 03:31:02.153913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:02.153851    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-096812-serving-cert\""
Oct 31 03:31:02.180144 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:02.180108    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"]
Oct 31 03:31:02.201059 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:02.201020    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbf932c9-3168-4564-aab2-5391dd41cc1a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-096812-856c94495f-zh5m8\" (UID: \"bbf932c9-3168-4564-aab2-5391dd41cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"
Oct 31 03:31:02.201226 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:02.201104    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqrt6\" (UniqueName: \"kubernetes.io/projected/bbf932c9-3168-4564-aab2-5391dd41cc1a-kube-api-access-mqrt6\") pod \"maas-api-e2e-worker-w4-096812-856c94495f-zh5m8\" (UID: \"bbf932c9-3168-4564-aab2-5391dd41cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"
Oct 31 03:31:02.201304 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:02.201292    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbf932c9-3168-4564-aab2-5391dd41cc1a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-096812-856c94495f-zh5m8\" (UID: \"bbf932c9-3168-4564-aab2-5391dd41cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"
Oct 31 03:31:02.201364 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:02.201318    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf932c9-3168-4564-aab2-5391dd41cc1a-metrics-certs\") pod \"maas-api-e2e-worker-w4-096812-856c94495f-zh5m8\" (UID: \"bbf932c9-3168-4564-aab2-5391dd41cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"
Oct 31 03:31:02.201450 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:31:02.201430    4447 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-serving-cert: secret "maas-api-e2e-worker-w4-096812-serving-cert" not found
Oct 31 03:31:02.201501 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:31:02.201489    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbf932c9-3168-4564-aab2-5391dd41cc1a-maas-api-tls podName:bbf932c9-3168-4564-aab2-5391dd41cc1a nodeName:}" failed. No retries permitted until 2026-10-31 03:31:02.701473487 +0000 UTC m=+688.212455879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bbf932c9-3168-4564-aab2-5391dd41cc1a-maas-api-tls") pod "maas-api-e2e-worker-w4-096812-856c94495f-zh5m8" (UID: "bbf932c9-3168-4564-aab2-5391dd41cc1a") : secret "maas-api-e2e-worker-w4-096812-serving-cert" not found
Oct 31 03:31:02.201629 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:02.201608    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbf932c9-3168-4564-aab2-5391dd41cc1a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-096812-856c94495f-zh5m8\" (UID: \"bbf932c9-3168-4564-aab2-5391dd41cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"
Oct 31 03:31:02.203833 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:02.203811    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf932c9-3168-4564-aab2-5391dd41cc1a-metrics-certs\") pod \"maas-api-e2e-worker-w4-096812-856c94495f-zh5m8\" (UID: \"bbf932c9-3168-4564-aab2-5391dd41cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"
Oct 31 03:31:02.269363 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:02.269323    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqrt6\" (UniqueName: \"kubernetes.io/projected/bbf932c9-3168-4564-aab2-5391dd41cc1a-kube-api-access-mqrt6\") pod \"maas-api-e2e-worker-w4-096812-856c94495f-zh5m8\" (UID: \"bbf932c9-3168-4564-aab2-5391dd41cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"
Oct 31 03:31:02.709508 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:02.709476    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbf932c9-3168-4564-aab2-5391dd41cc1a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-096812-856c94495f-zh5m8\" (UID: \"bbf932c9-3168-4564-aab2-5391dd41cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"
Oct 31 03:31:02.711957 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:02.711938    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbf932c9-3168-4564-aab2-5391dd41cc1a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-096812-856c94495f-zh5m8\" (UID: \"bbf932c9-3168-4564-aab2-5391dd41cc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"
Oct 31 03:31:02.978835 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:02.978744    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"
Oct 31 03:31:03.356849 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:03.356821    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"]
Oct 31 03:31:03.357941 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:31:03.357912    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf932c9_3168_4564_aab2_5391dd41cc1a.slice/crio-d1637d182f49669215488ee731abbfedc7b797afc69b5e6a46ede28ee0a92982 WatchSource:0}: Error finding container d1637d182f49669215488ee731abbfedc7b797afc69b5e6a46ede28ee0a92982: Status 404 returned error can't find the container with id d1637d182f49669215488ee731abbfedc7b797afc69b5e6a46ede28ee0a92982
Oct 31 03:31:03.604997 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:03.604918    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8" event={"ID":"bbf932c9-3168-4564-aab2-5391dd41cc1a","Type":"ContainerStarted","Data":"d1637d182f49669215488ee731abbfedc7b797afc69b5e6a46ede28ee0a92982"}
Oct 31 03:31:04.531609 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:04.531579    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt"
Oct 31 03:31:04.614940 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:04.614873    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8" event={"ID":"bbf932c9-3168-4564-aab2-5391dd41cc1a","Type":"ContainerStarted","Data":"d497fa4f32bc8b32e49d06065d13e0184679023e91f0759059e8abee44420992"}
Oct 31 03:31:04.615092 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:04.615035    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"
Oct 31 03:31:04.767255 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:04.767198    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8" podStartSLOduration=1.925546679 podStartE2EDuration="2.767183585s" totalImagesPullingTime="841.636906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:04.765340552 +0000 UTC m=+690.276322929" watchObservedRunningTime="2026-10-31 03:31:04.767183585 +0000 UTC m=+690.278166008"
Oct 31 03:31:05.539391 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:05.539366    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp"
Oct 31 03:31:07.554876 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:07.554849    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk"
Oct 31 03:31:10.627707 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:31:10.627677    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"
Oct 31 03:31:19.002354 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:31:19.002314    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:31:49.003496 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:31:49.003455    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:32:19.004104 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:32:19.004020    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:32:25.770968 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:25.770931    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"]
Oct 31 03:32:25.775065 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:25.775047    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:32:25.849129 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:25.848567    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-home\") pod \"simulator-w4-096812-kserve-5d865c545-4hk7f\" (UID: \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\") " pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:32:25.849129 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:25.848652    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-dshm\") pod \"simulator-w4-096812-kserve-5d865c545-4hk7f\" (UID: \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\") " pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:32:25.849129 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:25.848691    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-model-cache\") pod \"simulator-w4-096812-kserve-5d865c545-4hk7f\" (UID: \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\") " pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:32:25.849129 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:25.848747    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-tmp-dir\") pod \"simulator-w4-096812-kserve-5d865c545-4hk7f\" (UID: \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\") " pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:32:25.849129 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:25.848849    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-tls-certs\") pod \"simulator-w4-096812-kserve-5d865c545-4hk7f\" (UID: \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\") " pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:32:25.849129 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:25.848981    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2wwq\" (UniqueName: \"kubernetes.io/projected/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-kube-api-access-g2wwq\") pod \"simulator-w4-096812-kserve-5d865c545-4hk7f\" (UID: \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\") " pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:32:25.879747 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:25.879714    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-096812\"/\"default-dockercfg-jm4bw\""
Oct 31 03:32:25.880002 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:25.879842    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-096812\"/\"openshift-service-ca.crt\""
Oct 31 03:32:25.880002 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:25.879872    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-096812\"/\"simulator-w4-096812-kserve-self-signed-certs\""
Oct 31 03:32:25.880002 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:25.879952    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-096812\"/\"kube-root-ca.crt\""
Oct 31 03:32:25.883105 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:25.883085    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"]
Oct 31 03:32:25.950354 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:25.950322    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-dshm\") pod \"simulator-w4-096812-kserve-5d865c545-4hk7f\" (UID: \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\") " pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:32:25.950354 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:25.950370    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-model-cache\") pod \"simulator-w4-096812-kserve-5d865c545-4hk7f\" (UID: \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\") " pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:32:25.950626 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:25.950419    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-tmp-dir\") pod \"simulator-w4-096812-kserve-5d865c545-4hk7f\" (UID: \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\") " pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:32:25.950626 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:25.950506    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-tls-certs\") pod \"simulator-w4-096812-kserve-5d865c545-4hk7f\" (UID: \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\") " pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:32:25.950626 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:25.950610    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2wwq\" (UniqueName: \"kubernetes.io/projected/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-kube-api-access-g2wwq\") pod \"simulator-w4-096812-kserve-5d865c545-4hk7f\" (UID: \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\") " pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:32:25.950837 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:25.950646    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-home\") pod \"simulator-w4-096812-kserve-5d865c545-4hk7f\" (UID: \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\") " pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:32:25.950837 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:25.950821    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-model-cache\") pod \"simulator-w4-096812-kserve-5d865c545-4hk7f\" (UID: \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\") " pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:32:25.950948 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:25.950931    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-tmp-dir\") pod \"simulator-w4-096812-kserve-5d865c545-4hk7f\" (UID: \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\") " pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:32:25.951050 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:25.951033    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-home\") pod \"simulator-w4-096812-kserve-5d865c545-4hk7f\" (UID: \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\") " pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:32:25.952856 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:25.952829    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-dshm\") pod \"simulator-w4-096812-kserve-5d865c545-4hk7f\" (UID: \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\") " pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:32:25.953159 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:25.953143    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-tls-certs\") pod \"simulator-w4-096812-kserve-5d865c545-4hk7f\" (UID: \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\") " pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:32:26.012072 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:26.012042    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2wwq\" (UniqueName: \"kubernetes.io/projected/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-kube-api-access-g2wwq\") pod \"simulator-w4-096812-kserve-5d865c545-4hk7f\" (UID: \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\") " pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:32:26.087970 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:26.087842    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:32:26.469559 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:26.469527    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"]
Oct 31 03:32:26.470386 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:32:26.470359    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1c77031_ee5b_4d25_9b0d_dc7a49aaa1a6.slice/crio-578bbbc6b884590cadb8e8bacde128b1cdebb5b88a52f505042d5eac30b9ac06 WatchSource:0}: Error finding container 578bbbc6b884590cadb8e8bacde128b1cdebb5b88a52f505042d5eac30b9ac06: Status 404 returned error can't find the container with id 578bbbc6b884590cadb8e8bacde128b1cdebb5b88a52f505042d5eac30b9ac06
Oct 31 03:32:27.176136 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:27.176096    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" event={"ID":"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6","Type":"ContainerStarted","Data":"54a2908d8f0e8c52dc5ffcc83ce6a5d6fd68f5c70d5cadf0526bb955457dcbd9"}
Oct 31 03:32:27.176136 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:27.176142    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" event={"ID":"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6","Type":"ContainerStarted","Data":"578bbbc6b884590cadb8e8bacde128b1cdebb5b88a52f505042d5eac30b9ac06"}
Oct 31 03:32:27.296800 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:27.296740    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" podStartSLOduration=2.296725796 podStartE2EDuration="2.296725796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:27.293056029 +0000 UTC m=+772.804038409" watchObservedRunningTime="2026-10-31 03:32:27.296725796 +0000 UTC m=+772.807708174"
Oct 31 03:32:28.088573 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:28.088538    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:32:33.387288 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.387238    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"]
Oct 31 03:32:33.392988 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.392956    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:32:33.449229 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.449194    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"]
Oct 31 03:32:33.494662 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.494618    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-14cb5a\"/\"simulator-w5-14cb5a-kserve-self-signed-certs\""
Oct 31 03:32:33.494662 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.494658    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-14cb5a\"/\"default-dockercfg-gncl9\""
Oct 31 03:32:33.495253 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.495233    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-14cb5a\"/\"kube-root-ca.crt\""
Oct 31 03:32:33.495320 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.495266    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-14cb5a\"/\"openshift-service-ca.crt\""
Oct 31 03:32:33.541745 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.541711    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-tmp-dir\") pod \"simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6\" (UID: \"6a9113fb-e914-442b-aed1-e6b16edb5e91\") " pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:32:33.541952 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.541775    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jsq6\" (UniqueName: \"kubernetes.io/projected/6a9113fb-e914-442b-aed1-e6b16edb5e91-kube-api-access-7jsq6\") pod \"simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6\" (UID: \"6a9113fb-e914-442b-aed1-e6b16edb5e91\") " pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:32:33.541952 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.541870    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-dshm\") pod \"simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6\" (UID: \"6a9113fb-e914-442b-aed1-e6b16edb5e91\") " pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:32:33.542073 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.542054    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-model-cache\") pod \"simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6\" (UID: \"6a9113fb-e914-442b-aed1-e6b16edb5e91\") " pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:32:33.542122 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.542103    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-home\") pod \"simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6\" (UID: \"6a9113fb-e914-442b-aed1-e6b16edb5e91\") " pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:32:33.542226 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.542205    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a9113fb-e914-442b-aed1-e6b16edb5e91-tls-certs\") pod \"simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6\" (UID: \"6a9113fb-e914-442b-aed1-e6b16edb5e91\") " pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:32:33.643749 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.643637    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-tmp-dir\") pod \"simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6\" (UID: \"6a9113fb-e914-442b-aed1-e6b16edb5e91\") " pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:32:33.643749 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.643707    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jsq6\" (UniqueName: \"kubernetes.io/projected/6a9113fb-e914-442b-aed1-e6b16edb5e91-kube-api-access-7jsq6\") pod \"simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6\" (UID: \"6a9113fb-e914-442b-aed1-e6b16edb5e91\") " pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:32:33.643749 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.643736    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-dshm\") pod \"simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6\" (UID: \"6a9113fb-e914-442b-aed1-e6b16edb5e91\") " pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:32:33.644108 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.643847    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-model-cache\") pod \"simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6\" (UID: \"6a9113fb-e914-442b-aed1-e6b16edb5e91\") " pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:32:33.644108 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.643928    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-home\") pod \"simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6\" (UID: \"6a9113fb-e914-442b-aed1-e6b16edb5e91\") " pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:32:33.644108 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.643973    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a9113fb-e914-442b-aed1-e6b16edb5e91-tls-certs\") pod \"simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6\" (UID: \"6a9113fb-e914-442b-aed1-e6b16edb5e91\") " pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:32:33.644268 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.644108    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-tmp-dir\") pod \"simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6\" (UID: \"6a9113fb-e914-442b-aed1-e6b16edb5e91\") " pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:32:33.644327 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.644289    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-home\") pod \"simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6\" (UID: \"6a9113fb-e914-442b-aed1-e6b16edb5e91\") " pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:32:33.644327 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.644293    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-model-cache\") pod \"simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6\" (UID: \"6a9113fb-e914-442b-aed1-e6b16edb5e91\") " pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:32:33.646530 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.646500    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-dshm\") pod \"simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6\" (UID: \"6a9113fb-e914-442b-aed1-e6b16edb5e91\") " pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:32:33.646872 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.646855    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a9113fb-e914-442b-aed1-e6b16edb5e91-tls-certs\") pod \"simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6\" (UID: \"6a9113fb-e914-442b-aed1-e6b16edb5e91\") " pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:32:33.707504 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:33.707469    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jsq6\" (UniqueName: \"kubernetes.io/projected/6a9113fb-e914-442b-aed1-e6b16edb5e91-kube-api-access-7jsq6\") pod \"simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6\" (UID: \"6a9113fb-e914-442b-aed1-e6b16edb5e91\") " pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:32:34.007616 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:34.007576    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:32:34.397971 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:34.397941    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"]
Oct 31 03:32:34.398652 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:32:34.398623    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a9113fb_e914_442b_aed1_e6b16edb5e91.slice/crio-8059bb561b6a83b03055284815ea744088853621647d13859bfc513bc37b9fcb WatchSource:0}: Error finding container 8059bb561b6a83b03055284815ea744088853621647d13859bfc513bc37b9fcb: Status 404 returned error can't find the container with id 8059bb561b6a83b03055284815ea744088853621647d13859bfc513bc37b9fcb
Oct 31 03:32:35.236282 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:35.236239    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" event={"ID":"6a9113fb-e914-442b-aed1-e6b16edb5e91","Type":"ContainerStarted","Data":"faee75b798078dc1e3949ea294e2ab8726173c012afa0d22b2d7c6f132484e6e"}
Oct 31 03:32:35.236282 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:35.236287    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" event={"ID":"6a9113fb-e914-442b-aed1-e6b16edb5e91","Type":"ContainerStarted","Data":"8059bb561b6a83b03055284815ea744088853621647d13859bfc513bc37b9fcb"}
Oct 31 03:32:35.354316 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:35.354254    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" podStartSLOduration=2.354238027 podStartE2EDuration="2.354238027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:35.353163622 +0000 UTC m=+780.864146038" watchObservedRunningTime="2026-10-31 03:32:35.354238027 +0000 UTC m=+780.865220404"
Oct 31 03:32:36.008634 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:36.008593    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:32:36.088834 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:36.088800    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:32:36.102050 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:36.102023    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:32:36.255072 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:36.255043    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:32:44.008254 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:44.008216    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:32:44.021276 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:44.021247    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:32:44.315096 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:32:44.315014    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:32:49.003010 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:32:49.002967    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:33:19.003655 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:33:19.003620    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:33:49.006035 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:33:49.005949    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:34:07.033990 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.033956    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"]
Oct 31 03:34:07.038494 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.038458    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.091718 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.091688    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-b63425-openshift-default-dockercfg-frs87\""
Oct 31 03:34:07.145856 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.145822    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"]
Oct 31 03:34:07.231044 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.230999    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29a47d5-b793-43c1-befc-b32b16e31114-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.231044 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.231043    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29a47d5-b793-43c1-befc-b32b16e31114-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.231285 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.231154    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.231285 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.231208    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29a47d5-b793-43c1-befc-b32b16e31114-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.231360 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.231306    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.231360 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.231334    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.231457 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.231374    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.231457 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.231432    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.231534 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.231490    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwgts\" (UniqueName: \"kubernetes.io/projected/c29a47d5-b793-43c1-befc-b32b16e31114-kube-api-access-wwgts\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.332872 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.332781    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.332872 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.332830    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29a47d5-b793-43c1-befc-b32b16e31114-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.332872 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.332867    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.333199 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.332913    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.333199 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.332956    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.333199 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.332986    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.333199 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.333049    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwgts\" (UniqueName: \"kubernetes.io/projected/c29a47d5-b793-43c1-befc-b32b16e31114-kube-api-access-wwgts\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.333199 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.333121    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29a47d5-b793-43c1-befc-b32b16e31114-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.333199 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.333145    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29a47d5-b793-43c1-befc-b32b16e31114-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.333485 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.333240    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.333485 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.333296    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.333485 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.333432    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.333637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.333574    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.333800 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.333780    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29a47d5-b793-43c1-befc-b32b16e31114-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.335396 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.335370    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.335667 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.335650    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29a47d5-b793-43c1-befc-b32b16e31114-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.395188 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.395151    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29a47d5-b793-43c1-befc-b32b16e31114-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.395366 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.395257    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwgts\" (UniqueName: \"kubernetes.io/projected/c29a47d5-b793-43c1-befc-b32b16e31114-kube-api-access-wwgts\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:07.655134 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:07.655056    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:08.146423 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:34:08.146391    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29a47d5_b793_43c1_befc_b32b16e31114.slice/crio-7cd930df61870eeb80720e2bedccf97598d828b8706f21e013c501bf7281dfeb WatchSource:0}: Error finding container 7cd930df61870eeb80720e2bedccf97598d828b8706f21e013c501bf7281dfeb: Status 404 returned error can't find the container with id 7cd930df61870eeb80720e2bedccf97598d828b8706f21e013c501bf7281dfeb
Oct 31 03:34:08.146855 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:08.146654    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"]
Oct 31 03:34:08.148764 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:08.148737    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:34:08.148833 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:08.148808    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:34:08.148869 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:08.148834    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:34:08.906963 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:08.906931    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg" event={"ID":"c29a47d5-b793-43c1-befc-b32b16e31114","Type":"ContainerStarted","Data":"903b2472e6a9e553e5dd85d59116f24c8e730caed53d6b112868f64b953f71c9"}
Oct 31 03:34:08.907149 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:08.906982    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg" event={"ID":"c29a47d5-b793-43c1-befc-b32b16e31114","Type":"ContainerStarted","Data":"7cd930df61870eeb80720e2bedccf97598d828b8706f21e013c501bf7281dfeb"}
Oct 31 03:34:09.129064 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:09.129012    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg" podStartSLOduration=3.12899948 podStartE2EDuration="3.12899948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:09.078854696 +0000 UTC m=+874.589837073" watchObservedRunningTime="2026-10-31 03:34:09.12899948 +0000 UTC m=+874.639981856"
Oct 31 03:34:09.656046 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:09.656010    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:09.657504 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:09.657481    4447 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 03:34:09.657577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:09.657539    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg" podUID="c29a47d5-b793-43c1-befc-b32b16e31114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 03:34:10.656308 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:10.656270    4447 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 03:34:10.656700 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:10.656330    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg" podUID="c29a47d5-b793-43c1-befc-b32b16e31114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 03:34:11.655659 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:11.655622    4447 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 03:34:11.655863 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:11.655680    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg" podUID="c29a47d5-b793-43c1-befc-b32b16e31114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 03:34:12.655974 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:12.655928    4447 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 03:34:12.656397 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:12.655994    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg" podUID="c29a47d5-b793-43c1-befc-b32b16e31114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 03:34:13.656105 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:13.656073    4447 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 03:34:13.656490 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:13.656121    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg" podUID="c29a47d5-b793-43c1-befc-b32b16e31114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 03:34:14.722319 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:14.722280    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:14.723124 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:14.723106    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:14.723683 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:14.723662    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:34:19.005903 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:34:19.005852    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:34:22.463674 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.463640    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"]
Oct 31 03:34:22.468043 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.468023    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.554375 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.554339    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57796b75-644c-40e6-9773-d143da5fb821-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.554559 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.554452    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.554559 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.554486    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57796b75-644c-40e6-9773-d143da5fb821-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.554559 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.554544    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.554668 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.554568    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.554702 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.554671    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.554702 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.554693    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w5nv\" (UniqueName: \"kubernetes.io/projected/57796b75-644c-40e6-9773-d143da5fb821-kube-api-access-8w5nv\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.554763 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.554711    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.554796 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.554785    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57796b75-644c-40e6-9773-d143da5fb821-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.571328 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.571297    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e00ad3-openshift-default-dockercfg-s9kbq\""
Oct 31 03:34:22.576069 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.576037    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"]
Oct 31 03:34:22.655978 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.655930    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.655978 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.655968    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57796b75-644c-40e6-9773-d143da5fb821-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.656239 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.656013    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.656239 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.656051    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.656239 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.656117    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.656239 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.656141    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w5nv\" (UniqueName: \"kubernetes.io/projected/57796b75-644c-40e6-9773-d143da5fb821-kube-api-access-8w5nv\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.656239 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.656167    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.656513 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.656360    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57796b75-644c-40e6-9773-d143da5fb821-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.656513 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.656427    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.656619 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.656524    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57796b75-644c-40e6-9773-d143da5fb821-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.656695 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.656676    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.656754 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.656717    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.657169 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.657133    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57796b75-644c-40e6-9773-d143da5fb821-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.657258 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.657236    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.658819 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.658796    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57796b75-644c-40e6-9773-d143da5fb821-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.658948 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.658922    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.717758 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.717666    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57796b75-644c-40e6-9773-d143da5fb821-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.717914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.717875    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w5nv\" (UniqueName: \"kubernetes.io/projected/57796b75-644c-40e6-9773-d143da5fb821-kube-api-access-8w5nv\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:22.782601 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:22.782569    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:23.244320 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:23.244289    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"]
Oct 31 03:34:23.244772 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:34:23.244748    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57796b75_644c_40e6_9773_d143da5fb821.slice/crio-7a3d8d0eee7bb82c816aa55d2c0fa86c1db8d70d3c9d2dc3d0262daa39bb06be WatchSource:0}: Error finding container 7a3d8d0eee7bb82c816aa55d2c0fa86c1db8d70d3c9d2dc3d0262daa39bb06be: Status 404 returned error can't find the container with id 7a3d8d0eee7bb82c816aa55d2c0fa86c1db8d70d3c9d2dc3d0262daa39bb06be
Oct 31 03:34:23.247687 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:23.247644    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:34:23.247802 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:23.247722    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:34:23.247802 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:23.247783    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:34:24.019851 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:24.019794    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8" event={"ID":"57796b75-644c-40e6-9773-d143da5fb821","Type":"ContainerStarted","Data":"18ed63cae172bd167174279633990c21a6512ee34c0c714cd21da00ebdcb2e65"}
Oct 31 03:34:24.020321 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:24.019869    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8" event={"ID":"57796b75-644c-40e6-9773-d143da5fb821","Type":"ContainerStarted","Data":"7a3d8d0eee7bb82c816aa55d2c0fa86c1db8d70d3c9d2dc3d0262daa39bb06be"}
Oct 31 03:34:24.244691 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:24.244540    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8" podStartSLOduration=2.244520139 podStartE2EDuration="2.244520139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:24.193566663 +0000 UTC m=+889.704549039" watchObservedRunningTime="2026-10-31 03:34:24.244520139 +0000 UTC m=+889.755502518"
Oct 31 03:34:24.795908 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:24.790281    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:24.799909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:24.797190    4447 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 03:34:24.799909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:24.797268    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8" podUID="57796b75-644c-40e6-9773-d143da5fb821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 03:34:25.794908 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:25.794329    4447 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 03:34:25.794908 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:25.794408    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8" podUID="57796b75-644c-40e6-9773-d143da5fb821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 03:34:26.796909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:26.788318    4447 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 03:34:26.796909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:26.788416    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8" podUID="57796b75-644c-40e6-9773-d143da5fb821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 03:34:27.799724 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:27.799687    4447 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 03:34:27.800190 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:27.799753    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8" podUID="57796b75-644c-40e6-9773-d143da5fb821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 03:34:28.797909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:28.786364    4447 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 03:34:28.797909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:28.786435    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8" podUID="57796b75-644c-40e6-9773-d143da5fb821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 03:34:29.895819 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:29.895776    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:29.896328 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:29.896212    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:29.897229 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:29.897207    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:34:30.133582 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:30.133546    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"]
Oct 31 03:34:30.133760 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:30.133739    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp" podUID="3a944162-110a-4337-a053-21947af7dca1" containerName="istio-proxy" containerID="cri-o://5007d072c6844b24e2d205ae46d9e9bfcecd86c76b7ec46c7c0d5c8571f10e00" gracePeriod=30
Oct 31 03:34:34.737598 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:34.737572    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 03:34:34.747014 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:34.746992    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 03:34:35.400994 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.400919    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:34:35.538416 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.538386    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-workload-socket\") pod \"3a944162-110a-4337-a053-21947af7dca1\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") "
Oct 31 03:34:35.538580 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.538542    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a944162-110a-4337-a053-21947af7dca1-istio-token\" (UniqueName: \"kubernetes.io/projected/3a944162-110a-4337-a053-21947af7dca1-istio-token\") pod \"3a944162-110a-4337-a053-21947af7dca1\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") "
Oct 31 03:34:35.538580 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.538570    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-workload-certs\") pod \"3a944162-110a-4337-a053-21947af7dca1\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") "
Oct 31 03:34:35.538676 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.538587    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a944162-110a-4337-a053-21947af7dca1-kube-api-access-2kj4z\" (UniqueName: \"kubernetes.io/projected/3a944162-110a-4337-a053-21947af7dca1-kube-api-access-2kj4z\") pod \"3a944162-110a-4337-a053-21947af7dca1\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") "
Oct 31 03:34:35.538676 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.538623    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-credential-socket\") pod \"3a944162-110a-4337-a053-21947af7dca1\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") "
Oct 31 03:34:35.538676 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.538648    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a944162-110a-4337-a053-21947af7dca1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a944162-110a-4337-a053-21947af7dca1-istiod-ca-cert\") pod \"3a944162-110a-4337-a053-21947af7dca1\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") "
Oct 31 03:34:35.538823 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.538679    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-workload-socket" pod "3a944162-110a-4337-a053-21947af7dca1" (UID: "3a944162-110a-4337-a053-21947af7dca1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:35.538823 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.538696    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a944162-110a-4337-a053-21947af7dca1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a944162-110a-4337-a053-21947af7dca1-istio-podinfo\") pod \"3a944162-110a-4337-a053-21947af7dca1\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") "
Oct 31 03:34:35.538823 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.538745    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-istio-data\") pod \"3a944162-110a-4337-a053-21947af7dca1\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") "
Oct 31 03:34:35.538823 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.538792    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-istio-envoy\") pod \"3a944162-110a-4337-a053-21947af7dca1\" (UID: \"3a944162-110a-4337-a053-21947af7dca1\") "
Oct 31 03:34:35.539051 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.538899    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-workload-certs" pod "3a944162-110a-4337-a053-21947af7dca1" (UID: "3a944162-110a-4337-a053-21947af7dca1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:35.539051 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.538968    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-credential-socket" pod "3a944162-110a-4337-a053-21947af7dca1" (UID: "3a944162-110a-4337-a053-21947af7dca1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:35.539527 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.539504    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:34:35.539655 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.539530    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:34:35.539655 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.539543    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:34:35.540137 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.540102    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-istio-data" pod "3a944162-110a-4337-a053-21947af7dca1" (UID: "3a944162-110a-4337-a053-21947af7dca1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:35.541480 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.541454    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a944162-110a-4337-a053-21947af7dca1-istiod-ca-cert" pod "3a944162-110a-4337-a053-21947af7dca1" (UID: "3a944162-110a-4337-a053-21947af7dca1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:34:35.544421 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.544380    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a944162-110a-4337-a053-21947af7dca1-istio-token" pod "3a944162-110a-4337-a053-21947af7dca1" (UID: "3a944162-110a-4337-a053-21947af7dca1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:34:35.544529 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.544432    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-istio-envoy" pod "3a944162-110a-4337-a053-21947af7dca1" (UID: "3a944162-110a-4337-a053-21947af7dca1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:35.545022 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.544996    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a944162-110a-4337-a053-21947af7dca1-kube-api-access-2kj4z" pod "3a944162-110a-4337-a053-21947af7dca1" (UID: "3a944162-110a-4337-a053-21947af7dca1"). InnerVolumeSpecName "kube-api-access-2kj4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:34:35.545225 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.545201    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a944162-110a-4337-a053-21947af7dca1-istio-podinfo" pod "3a944162-110a-4337-a053-21947af7dca1" (UID: "3a944162-110a-4337-a053-21947af7dca1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:34:35.640598 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.640555    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a944162-110a-4337-a053-21947af7dca1-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:34:35.640598 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.640590    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kj4z\" (UniqueName: \"kubernetes.io/projected/3a944162-110a-4337-a053-21947af7dca1-kube-api-access-2kj4z\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:34:35.640598 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.640600    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a944162-110a-4337-a053-21947af7dca1-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:34:35.640598 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.640609    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a944162-110a-4337-a053-21947af7dca1-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:34:35.640926 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.640618    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:34:35.640926 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:35.640627    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a944162-110a-4337-a053-21947af7dca1-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:34:36.103158 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:36.103119    4447 generic.go:356] "Generic (PLEG): container finished" podID="3a944162-110a-4337-a053-21947af7dca1" containerID="5007d072c6844b24e2d205ae46d9e9bfcecd86c76b7ec46c7c0d5c8571f10e00" exitCode=0
Oct 31 03:34:36.103599 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:36.103178    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"
Oct 31 03:34:36.103599 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:36.103200    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp" event={"ID":"3a944162-110a-4337-a053-21947af7dca1","Type":"ContainerDied","Data":"5007d072c6844b24e2d205ae46d9e9bfcecd86c76b7ec46c7c0d5c8571f10e00"}
Oct 31 03:34:36.103599 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:36.103284    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp" event={"ID":"3a944162-110a-4337-a053-21947af7dca1","Type":"ContainerDied","Data":"4ba6f729a36061427162f6f1f4eec862610fb31647fe7763cf593e456a99cdca"}
Oct 31 03:34:36.103599 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:36.103314    4447 scope.go:122] "RemoveContainer" containerID="5007d072c6844b24e2d205ae46d9e9bfcecd86c76b7ec46c7c0d5c8571f10e00"
Oct 31 03:34:36.114996 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:36.114974    4447 scope.go:122] "RemoveContainer" containerID="5007d072c6844b24e2d205ae46d9e9bfcecd86c76b7ec46c7c0d5c8571f10e00"
Oct 31 03:34:36.115314 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:34:36.115291    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5007d072c6844b24e2d205ae46d9e9bfcecd86c76b7ec46c7c0d5c8571f10e00\": container with ID starting with 5007d072c6844b24e2d205ae46d9e9bfcecd86c76b7ec46c7c0d5c8571f10e00 not found: ID does not exist" containerID="5007d072c6844b24e2d205ae46d9e9bfcecd86c76b7ec46c7c0d5c8571f10e00"
Oct 31 03:34:36.115382 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:36.115328    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5007d072c6844b24e2d205ae46d9e9bfcecd86c76b7ec46c7c0d5c8571f10e00"} err="failed to get container status \"5007d072c6844b24e2d205ae46d9e9bfcecd86c76b7ec46c7c0d5c8571f10e00\": rpc error: code = NotFound desc = could not find container \"5007d072c6844b24e2d205ae46d9e9bfcecd86c76b7ec46c7c0d5c8571f10e00\": container with ID starting with 5007d072c6844b24e2d205ae46d9e9bfcecd86c76b7ec46c7c0d5c8571f10e00 not found: ID does not exist"
Oct 31 03:34:36.323190 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:36.323145    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"]
Oct 31 03:34:36.327424 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:36.327393    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jncxp"]
Oct 31 03:34:36.609188 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:34:36.609159    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a944162-110a-4337-a053-21947af7dca1" path="/var/lib/kubelet/pods/3a944162-110a-4337-a053-21947af7dca1/volumes"
Oct 31 03:34:49.003366 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:34:49.003329    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:35:08.822622 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:08.822588    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"]
Oct 31 03:35:08.823491 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:08.823462    4447 cpu_manager.go:414] "containerMap: removing container" podUID="3a944162-110a-4337-a053-21947af7dca1" containerName="istio-proxy"
Oct 31 03:35:08.823491 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:08.823482    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a944162-110a-4337-a053-21947af7dca1" containerName="istio-proxy"
Oct 31 03:35:08.823859 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:08.823667    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a944162-110a-4337-a053-21947af7dca1" containerName="istio-proxy"
Oct 31 03:35:08.827433 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:08.827415    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:08.879969 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:08.879940    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-dec4a0-openshift-default-dockercfg-5rrgk\""
Oct 31 03:35:08.933205 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:08.933175    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"]
Oct 31 03:35:08.946015 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:08.945979    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:08.946273 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:08.946247    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:08.946397 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:08.946289    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:08.946448 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:08.946401    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:08.946448 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:08.946437    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:08.946537 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:08.946457    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:08.946622 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:08.946605    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:08.946690 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:08.946674    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm7db\" (UniqueName: \"kubernetes.io/projected/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-kube-api-access-tm7db\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:08.946747 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:08.946734    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:09.048464 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:09.048423    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:09.048640 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:09.048489    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:09.048640 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:09.048532    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:09.048640 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:09.048557    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:09.048776 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:09.048654    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:09.048776 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:09.048729    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm7db\" (UniqueName: \"kubernetes.io/projected/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-kube-api-access-tm7db\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:09.048878 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:09.048790    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:09.048878 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:09.048848    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:09.049060 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:09.048925    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:09.049060 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:09.048935    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:09.049060 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:09.048997    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:09.049217 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:09.049204    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:09.049299 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:09.049280    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:09.049511 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:09.049494    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:09.050945 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:09.050924    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:09.051233 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:09.051213    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:09.110240 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:09.110160    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:09.110240 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:09.110225    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm7db\" (UniqueName: \"kubernetes.io/projected/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-kube-api-access-tm7db\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:09.141349 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:09.141312    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:09.585136 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:09.585110    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"]
Oct 31 03:35:09.586731 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:35:09.586704    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0926be7e_f12d_474d_a3c8_1a3eb3fbc43f.slice/crio-17bd92b40f86a927b26b4a8b29768cc4ea2d8700376f6cadca49a1eb7d232493 WatchSource:0}: Error finding container 17bd92b40f86a927b26b4a8b29768cc4ea2d8700376f6cadca49a1eb7d232493: Status 404 returned error can't find the container with id 17bd92b40f86a927b26b4a8b29768cc4ea2d8700376f6cadca49a1eb7d232493
Oct 31 03:35:09.589152 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:09.589120    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:35:09.589240 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:09.589189    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:35:09.589240 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:09.589221    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:35:10.341002 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:10.340952    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q" event={"ID":"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f","Type":"ContainerStarted","Data":"aa90022d05a77afcadb5f852f4aa8d24d57b78805cfeb9c03ac6cd7d3bfe2b50"}
Oct 31 03:35:10.341395 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:10.341019    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q" event={"ID":"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f","Type":"ContainerStarted","Data":"17bd92b40f86a927b26b4a8b29768cc4ea2d8700376f6cadca49a1eb7d232493"}
Oct 31 03:35:10.512613 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:10.512559    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q" podStartSLOduration=2.512541695 podStartE2EDuration="2.512541695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:10.511142863 +0000 UTC m=+936.022125237" watchObservedRunningTime="2026-10-31 03:35:10.512541695 +0000 UTC m=+936.023524073"
Oct 31 03:35:11.141483 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:11.141450    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:11.142959 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:11.142935    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 03:35:11.143078 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:11.142980    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q" podUID="0926be7e-f12d-474d-a3c8-1a3eb3fbc43f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 03:35:12.142304 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:12.142270    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 03:35:12.142689 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:12.142320    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q" podUID="0926be7e-f12d-474d-a3c8-1a3eb3fbc43f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 03:35:13.142139 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:13.142102    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 03:35:13.142301 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:13.142159    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q" podUID="0926be7e-f12d-474d-a3c8-1a3eb3fbc43f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 03:35:14.142003 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:14.141964    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 03:35:14.142417 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:14.142021    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q" podUID="0926be7e-f12d-474d-a3c8-1a3eb3fbc43f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 03:35:15.141655 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:15.141618    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 03:35:15.141849 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:15.141671    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q" podUID="0926be7e-f12d-474d-a3c8-1a3eb3fbc43f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 03:35:16.142643 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:16.142605    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 03:35:16.143205 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:16.142685    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q" podUID="0926be7e-f12d-474d-a3c8-1a3eb3fbc43f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 03:35:17.146227 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:17.146194    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:17.146645 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:17.146564    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:17.147369 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:17.147351    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:35:19.007041 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:35:19.007004    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:35:27.262116 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:27.262065    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c11ccea-sqgjs"]
Oct 31 03:35:27.272329 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:27.272296    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c11ccea-sqgjs"
Oct 31 03:35:27.321631 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:27.321584    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c11ccea-sqgjs"]
Oct 31 03:35:27.366782 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:27.366750    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:35:27.366981 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:27.366790    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 03:35:27.393094 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:27.393059    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ad860c-f831-4582-9f6b-700c1edf2c67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c11ccea-sqgjs\" (UID: \"18ad860c-f831-4582-9f6b-700c1edf2c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c11ccea-sqgjs"
Oct 31 03:35:27.393094 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:27.393101    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ad860c-f831-4582-9f6b-700c1edf2c67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c11ccea-sqgjs\" (UID: \"18ad860c-f831-4582-9f6b-700c1edf2c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c11ccea-sqgjs"
Oct 31 03:35:27.393409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:27.393137    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68ck4\" (UniqueName: \"kubernetes.io/projected/18ad860c-f831-4582-9f6b-700c1edf2c67-kube-api-access-68ck4\") pod \"maas-api-revoke-sub-3c11ccea-sqgjs\" (UID: \"18ad860c-f831-4582-9f6b-700c1edf2c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c11ccea-sqgjs"
Oct 31 03:35:27.494647 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:27.494610    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ad860c-f831-4582-9f6b-700c1edf2c67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c11ccea-sqgjs\" (UID: \"18ad860c-f831-4582-9f6b-700c1edf2c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c11ccea-sqgjs"
Oct 31 03:35:27.494647 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:27.494648    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ad860c-f831-4582-9f6b-700c1edf2c67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c11ccea-sqgjs\" (UID: \"18ad860c-f831-4582-9f6b-700c1edf2c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c11ccea-sqgjs"
Oct 31 03:35:27.494927 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:27.494685    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68ck4\" (UniqueName: \"kubernetes.io/projected/18ad860c-f831-4582-9f6b-700c1edf2c67-kube-api-access-68ck4\") pod \"maas-api-revoke-sub-3c11ccea-sqgjs\" (UID: \"18ad860c-f831-4582-9f6b-700c1edf2c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c11ccea-sqgjs"
Oct 31 03:35:27.495304 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:27.495279    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ad860c-f831-4582-9f6b-700c1edf2c67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c11ccea-sqgjs\" (UID: \"18ad860c-f831-4582-9f6b-700c1edf2c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c11ccea-sqgjs"
Oct 31 03:35:27.497366 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:27.497339    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ad860c-f831-4582-9f6b-700c1edf2c67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c11ccea-sqgjs\" (UID: \"18ad860c-f831-4582-9f6b-700c1edf2c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c11ccea-sqgjs"
Oct 31 03:35:27.556380 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:27.556305    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68ck4\" (UniqueName: \"kubernetes.io/projected/18ad860c-f831-4582-9f6b-700c1edf2c67-kube-api-access-68ck4\") pod \"maas-api-revoke-sub-3c11ccea-sqgjs\" (UID: \"18ad860c-f831-4582-9f6b-700c1edf2c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c11ccea-sqgjs"
Oct 31 03:35:27.601513 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:27.601475    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c11ccea-sqgjs"
Oct 31 03:35:28.035000 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:28.034973    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c11ccea-sqgjs"]
Oct 31 03:35:28.493591 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:28.493564    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c11ccea-sqgjs" event={"ID":"18ad860c-f831-4582-9f6b-700c1edf2c67","Type":"ContainerStarted","Data":"36f198c2126047bdff367fc2c8c4b54da39c3cca6709c8f8225b7133771e2c0e"}
Oct 31 03:35:29.502415 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:29.502383    4447 generic.go:356] "Generic (PLEG): container finished" podID="18ad860c-f831-4582-9f6b-700c1edf2c67" containerID="d4944093c2351ad1f06314708a20c6b89f7d5147ae97e38f240024ad6b380acb" exitCode=0
Oct 31 03:35:29.502799 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:29.502468    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c11ccea-sqgjs" event={"ID":"18ad860c-f831-4582-9f6b-700c1edf2c67","Type":"ContainerDied","Data":"d4944093c2351ad1f06314708a20c6b89f7d5147ae97e38f240024ad6b380acb"}
Oct 31 03:35:30.664710 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:30.664678    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c11ccea-sqgjs"
Oct 31 03:35:30.842899 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:30.842790    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ad860c-f831-4582-9f6b-700c1edf2c67-kube-api-access-68ck4\" (UniqueName: \"kubernetes.io/projected/18ad860c-f831-4582-9f6b-700c1edf2c67-kube-api-access-68ck4\") pod \"18ad860c-f831-4582-9f6b-700c1edf2c67\" (UID: \"18ad860c-f831-4582-9f6b-700c1edf2c67\") "
Oct 31 03:35:30.842899 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:30.842837    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18ad860c-f831-4582-9f6b-700c1edf2c67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ad860c-f831-4582-9f6b-700c1edf2c67-maas-api-xks-ca\") pod \"18ad860c-f831-4582-9f6b-700c1edf2c67\" (UID: \"18ad860c-f831-4582-9f6b-700c1edf2c67\") "
Oct 31 03:35:30.842899 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:30.842860    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18ad860c-f831-4582-9f6b-700c1edf2c67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ad860c-f831-4582-9f6b-700c1edf2c67-maas-api-openshift-ca\") pod \"18ad860c-f831-4582-9f6b-700c1edf2c67\" (UID: \"18ad860c-f831-4582-9f6b-700c1edf2c67\") "
Oct 31 03:35:30.845297 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:30.845267    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ad860c-f831-4582-9f6b-700c1edf2c67-maas-api-openshift-ca" pod "18ad860c-f831-4582-9f6b-700c1edf2c67" (UID: "18ad860c-f831-4582-9f6b-700c1edf2c67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:35:30.847090 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:30.847068    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ad860c-f831-4582-9f6b-700c1edf2c67-maas-api-xks-ca" pod "18ad860c-f831-4582-9f6b-700c1edf2c67" (UID: "18ad860c-f831-4582-9f6b-700c1edf2c67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:35:30.847202 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:30.847150    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ad860c-f831-4582-9f6b-700c1edf2c67-kube-api-access-68ck4" pod "18ad860c-f831-4582-9f6b-700c1edf2c67" (UID: "18ad860c-f831-4582-9f6b-700c1edf2c67"). InnerVolumeSpecName "kube-api-access-68ck4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:35:30.944832 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:30.944792    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ad860c-f831-4582-9f6b-700c1edf2c67-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:35:30.944832 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:30.944827    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68ck4\" (UniqueName: \"kubernetes.io/projected/18ad860c-f831-4582-9f6b-700c1edf2c67-kube-api-access-68ck4\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:35:30.944832 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:30.944837    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ad860c-f831-4582-9f6b-700c1edf2c67-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:35:31.517474 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:31.517444    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c11ccea-sqgjs"
Oct 31 03:35:31.517665 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:31.517490    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c11ccea-sqgjs" event={"ID":"18ad860c-f831-4582-9f6b-700c1edf2c67","Type":"ContainerDied","Data":"36f198c2126047bdff367fc2c8c4b54da39c3cca6709c8f8225b7133771e2c0e"}
Oct 31 03:35:31.517665 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:31.517546    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36f198c2126047bdff367fc2c8c4b54da39c3cca6709c8f8225b7133771e2c0e"
Oct 31 03:35:49.015189 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:35:49.015144    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:35:53.922195 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:53.922165    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"]
Oct 31 03:35:53.922837 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:53.922823    4447 cpu_manager.go:414] "containerMap: removing container" podUID="18ad860c-f831-4582-9f6b-700c1edf2c67" containerName="revoke-keys"
Oct 31 03:35:53.922899 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:53.922839    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ad860c-f831-4582-9f6b-700c1edf2c67" containerName="revoke-keys"
Oct 31 03:35:53.923011 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:53.923000    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ad860c-f831-4582-9f6b-700c1edf2c67" containerName="revoke-keys"
Oct 31 03:35:53.926876 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:53.926856    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:53.982024 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:53.981989    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b98893-openshift-default-dockercfg-f7nhz\""
Oct 31 03:35:54.029359 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.029321    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.029559 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.029375    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.029559 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.029430    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.029559 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.029524    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.029559 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.029554    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qqkr\" (UniqueName: \"kubernetes.io/projected/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-kube-api-access-5qqkr\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.029791 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.029588    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.029791 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.029636    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.029791 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.029662    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.029791 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.029758    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.037650 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.037618    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"]
Oct 31 03:35:54.131247 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.131203    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.131453 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.131305    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.131453 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.131336    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.131453 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.131382    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.131620 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.131469    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.131620 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.131501    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qqkr\" (UniqueName: \"kubernetes.io/projected/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-kube-api-access-5qqkr\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.131620 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.131538    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.131783 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.131692    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.131783 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.131708    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.131783 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.131700    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.131783 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.131760    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.132045 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.131949    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.132096 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.132076    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.132324 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.132228    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.134017 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.133991    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.134328 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.134306    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.194383 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.194306    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.194525 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.194426    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qqkr\" (UniqueName: \"kubernetes.io/projected/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-kube-api-access-5qqkr\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.240410 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.240380    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:54.673275 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.673244    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"]
Oct 31 03:35:54.674239 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:35:54.674205    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d2a84e_6aad_4669_a456_ebe20cf2f0e9.slice/crio-255b21e0bf99a4ca1bd78b9f731f6b0170b0d5e8ed1bc05b5f537f9aae0608ef WatchSource:0}: Error finding container 255b21e0bf99a4ca1bd78b9f731f6b0170b0d5e8ed1bc05b5f537f9aae0608ef: Status 404 returned error can't find the container with id 255b21e0bf99a4ca1bd78b9f731f6b0170b0d5e8ed1bc05b5f537f9aae0608ef
Oct 31 03:35:54.677068 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.677030    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:35:54.677161 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.677116    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:35:54.677161 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.677156    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:35:54.684846 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:54.684816    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd" event={"ID":"24d2a84e-6aad-4669-a456-ebe20cf2f0e9","Type":"ContainerStarted","Data":"255b21e0bf99a4ca1bd78b9f731f6b0170b0d5e8ed1bc05b5f537f9aae0608ef"}
Oct 31 03:35:55.701386 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:55.701347    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd" event={"ID":"24d2a84e-6aad-4669-a456-ebe20cf2f0e9","Type":"ContainerStarted","Data":"edc9f8388d56defcc6825e96a1fdd571b057cfbf9ebe3aecf80a184b2532f7ee"}
Oct 31 03:35:55.873955 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:55.873861    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd" podStartSLOduration=2.873842149 podStartE2EDuration="2.873842149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:55.873012172 +0000 UTC m=+981.383994550" watchObservedRunningTime="2026-10-31 03:35:55.873842149 +0000 UTC m=+981.384824527"
Oct 31 03:35:56.240673 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:56.240632    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:35:56.242763 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:56.242730    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 03:35:56.242951 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:56.242790    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd" podUID="24d2a84e-6aad-4669-a456-ebe20cf2f0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 03:35:57.240816 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:57.240778    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 03:35:57.241262 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:57.240845    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd" podUID="24d2a84e-6aad-4669-a456-ebe20cf2f0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 03:35:58.241633 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:58.241594    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 03:35:58.242053 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:58.241660    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd" podUID="24d2a84e-6aad-4669-a456-ebe20cf2f0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 03:35:59.240979 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:59.240935    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 03:35:59.241188 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:35:59.241018    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd" podUID="24d2a84e-6aad-4669-a456-ebe20cf2f0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 03:36:00.241621 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:00.241563    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 03:36:00.242213 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:00.241663    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd" podUID="24d2a84e-6aad-4669-a456-ebe20cf2f0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 03:36:01.344427 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:01.344393    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:36:01.345195 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:01.345172    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:36:01.345995 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:01.345975    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:36:09.590640 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.590609    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"]
Oct 31 03:36:09.594914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.594878    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.641957 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.641925    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.642116 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.641968    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.642116 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.642065    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.642195 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.642155    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.642233 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.642194    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mcsp\" (UniqueName: \"kubernetes.io/projected/8954a5a4-2ec5-4443-adef-155c6781cb89-kube-api-access-7mcsp\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.642378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.642358    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.642417 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.642402    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.642486 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.642465    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.642566 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.642553    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8954a5a4-2ec5-4443-adef-155c6781cb89-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.647732 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.647706    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-14cb5a-openshift-default-dockercfg-hg7j6\""
Oct 31 03:36:09.704416 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.704377    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"]
Oct 31 03:36:09.743811 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.743776    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.744074 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.743925    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mcsp\" (UniqueName: \"kubernetes.io/projected/8954a5a4-2ec5-4443-adef-155c6781cb89-kube-api-access-7mcsp\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.744074 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.744027    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.744074 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.744061    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.744247 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.744089    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.744247 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.744181    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8954a5a4-2ec5-4443-adef-155c6781cb89-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.744353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.744244    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.744353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.744278    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.744353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.744329    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.745188 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.745163    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8954a5a4-2ec5-4443-adef-155c6781cb89-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.745727 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.745708    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.745861 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.745790    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.747326 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.747303    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.747448 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.747368    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.748109 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.748085    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.749170 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.749146    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.805352 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.805327    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.805514 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.805459    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mcsp\" (UniqueName: \"kubernetes.io/projected/8954a5a4-2ec5-4443-adef-155c6781cb89-kube-api-access-7mcsp\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:09.910820 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:09.910689    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:10.356812 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:10.356561    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"]
Oct 31 03:36:10.357979 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:36:10.357955    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8954a5a4_2ec5_4443_adef_155c6781cb89.slice/crio-1479185e901bda3c3b6040fb9b2863cb7e1786b469e12f8f5e97b516d4ece922 WatchSource:0}: Error finding container 1479185e901bda3c3b6040fb9b2863cb7e1786b469e12f8f5e97b516d4ece922: Status 404 returned error can't find the container with id 1479185e901bda3c3b6040fb9b2863cb7e1786b469e12f8f5e97b516d4ece922
Oct 31 03:36:10.360475 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:10.360442    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:36:10.360584 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:10.360521    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:36:10.360584 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:10.360564    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:36:10.807756 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:10.807717    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m" event={"ID":"8954a5a4-2ec5-4443-adef-155c6781cb89","Type":"ContainerStarted","Data":"4846319f4401a761790dee40aa327c47d9374d74ee3f16042483df9ef8fe3760"}
Oct 31 03:36:10.810340 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:10.807783    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m" event={"ID":"8954a5a4-2ec5-4443-adef-155c6781cb89","Type":"ContainerStarted","Data":"1479185e901bda3c3b6040fb9b2863cb7e1786b469e12f8f5e97b516d4ece922"}
Oct 31 03:36:10.911497 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:10.911468    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:10.929020 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:10.928952    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m" podStartSLOduration=1.928933842 podStartE2EDuration="1.928933842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:10.928597489 +0000 UTC m=+996.439579892" watchObservedRunningTime="2026-10-31 03:36:10.928933842 +0000 UTC m=+996.439916221"
Oct 31 03:36:11.912786 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:11.912752    4447 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 03:36:11.913203 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:11.912814    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m" podUID="8954a5a4-2ec5-4443-adef-155c6781cb89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 03:36:12.912056 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:12.912018    4447 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 03:36:12.912244 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:12.912078    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m" podUID="8954a5a4-2ec5-4443-adef-155c6781cb89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 03:36:13.911458 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:13.911421    4447 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 03:36:13.911921 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:13.911479    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m" podUID="8954a5a4-2ec5-4443-adef-155c6781cb89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 03:36:14.912010 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:14.911968    4447 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 03:36:14.912416 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:14.912042    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m" podUID="8954a5a4-2ec5-4443-adef-155c6781cb89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 03:36:15.912000 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:15.911968    4447 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 03:36:15.912207 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:15.912024    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m" podUID="8954a5a4-2ec5-4443-adef-155c6781cb89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 03:36:17.094677 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:17.094638    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:17.095164 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:17.094990    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:17.096046 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:17.096022    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:36:19.003703 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:36:19.003666    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:36:33.900791 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:33.900757    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"]
Oct 31 03:36:33.905187 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:33.905171    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:36:33.987347 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:33.987315    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-model-cache\") pod \"distinct-w0-909365-kserve-85796b5977-68h48\" (UID: \"00e44046-fb2c-481f-8221-190358d77d1a\") " pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:36:33.987347 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:33.987360    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-dshm\") pod \"distinct-w0-909365-kserve-85796b5977-68h48\" (UID: \"00e44046-fb2c-481f-8221-190358d77d1a\") " pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:36:33.987602 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:33.987390    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-tmp-dir\") pod \"distinct-w0-909365-kserve-85796b5977-68h48\" (UID: \"00e44046-fb2c-481f-8221-190358d77d1a\") " pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:36:33.987602 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:33.987506    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e44046-fb2c-481f-8221-190358d77d1a-tls-certs\") pod \"distinct-w0-909365-kserve-85796b5977-68h48\" (UID: \"00e44046-fb2c-481f-8221-190358d77d1a\") " pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:36:33.987602 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:33.987556    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-home\") pod \"distinct-w0-909365-kserve-85796b5977-68h48\" (UID: \"00e44046-fb2c-481f-8221-190358d77d1a\") " pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:36:33.987725 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:33.987669    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfppc\" (UniqueName: \"kubernetes.io/projected/00e44046-fb2c-481f-8221-190358d77d1a-kube-api-access-vfppc\") pod \"distinct-w0-909365-kserve-85796b5977-68h48\" (UID: \"00e44046-fb2c-481f-8221-190358d77d1a\") " pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:36:34.008768 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:34.008732    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-909365\"/\"kube-root-ca.crt\""
Oct 31 03:36:34.008768 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:34.008734    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-909365\"/\"default-dockercfg-s5hcs\""
Oct 31 03:36:34.009166 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:34.009140    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-909365\"/\"distinct-w0-909365-kserve-self-signed-certs\""
Oct 31 03:36:34.009304 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:34.009169    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-909365\"/\"openshift-service-ca.crt\""
Oct 31 03:36:34.013261 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:34.013226    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"]
Oct 31 03:36:34.088971 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:34.088936    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-tmp-dir\") pod \"distinct-w0-909365-kserve-85796b5977-68h48\" (UID: \"00e44046-fb2c-481f-8221-190358d77d1a\") " pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:36:34.089174 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:34.088981    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e44046-fb2c-481f-8221-190358d77d1a-tls-certs\") pod \"distinct-w0-909365-kserve-85796b5977-68h48\" (UID: \"00e44046-fb2c-481f-8221-190358d77d1a\") " pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:36:34.089174 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:34.089025    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-home\") pod \"distinct-w0-909365-kserve-85796b5977-68h48\" (UID: \"00e44046-fb2c-481f-8221-190358d77d1a\") " pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:36:34.089174 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:34.089098    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfppc\" (UniqueName: \"kubernetes.io/projected/00e44046-fb2c-481f-8221-190358d77d1a-kube-api-access-vfppc\") pod \"distinct-w0-909365-kserve-85796b5977-68h48\" (UID: \"00e44046-fb2c-481f-8221-190358d77d1a\") " pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:36:34.089360 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:34.089195    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-model-cache\") pod \"distinct-w0-909365-kserve-85796b5977-68h48\" (UID: \"00e44046-fb2c-481f-8221-190358d77d1a\") " pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:36:34.089360 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:34.089227    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-dshm\") pod \"distinct-w0-909365-kserve-85796b5977-68h48\" (UID: \"00e44046-fb2c-481f-8221-190358d77d1a\") " pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:36:34.089360 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:34.089348    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-tmp-dir\") pod \"distinct-w0-909365-kserve-85796b5977-68h48\" (UID: \"00e44046-fb2c-481f-8221-190358d77d1a\") " pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:36:34.089529 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:34.089508    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-model-cache\") pod \"distinct-w0-909365-kserve-85796b5977-68h48\" (UID: \"00e44046-fb2c-481f-8221-190358d77d1a\") " pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:36:34.089529 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:34.089518    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-home\") pod \"distinct-w0-909365-kserve-85796b5977-68h48\" (UID: \"00e44046-fb2c-481f-8221-190358d77d1a\") " pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:36:34.091515 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:34.091486    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-dshm\") pod \"distinct-w0-909365-kserve-85796b5977-68h48\" (UID: \"00e44046-fb2c-481f-8221-190358d77d1a\") " pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:36:34.091765 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:34.091751    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e44046-fb2c-481f-8221-190358d77d1a-tls-certs\") pod \"distinct-w0-909365-kserve-85796b5977-68h48\" (UID: \"00e44046-fb2c-481f-8221-190358d77d1a\") " pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:36:34.153402 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:34.153320    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfppc\" (UniqueName: \"kubernetes.io/projected/00e44046-fb2c-481f-8221-190358d77d1a-kube-api-access-vfppc\") pod \"distinct-w0-909365-kserve-85796b5977-68h48\" (UID: \"00e44046-fb2c-481f-8221-190358d77d1a\") " pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:36:34.218502 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:34.218468    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:36:34.606780 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:36:34.606748    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00e44046_fb2c_481f_8221_190358d77d1a.slice/crio-978df90d85f49d0a65ca74dab3da2f2e4c73620704c7f3027dafa70e9a4d5575 WatchSource:0}: Error finding container 978df90d85f49d0a65ca74dab3da2f2e4c73620704c7f3027dafa70e9a4d5575: Status 404 returned error can't find the container with id 978df90d85f49d0a65ca74dab3da2f2e4c73620704c7f3027dafa70e9a4d5575
Oct 31 03:36:34.611746 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:34.611725    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"]
Oct 31 03:36:34.981204 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:34.981164    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" event={"ID":"00e44046-fb2c-481f-8221-190358d77d1a","Type":"ContainerStarted","Data":"c1b9ff58d3120c3442e2c679dabdab4d79993a84610858c8be1287bd4103a006"}
Oct 31 03:36:34.981657 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:34.981216    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" event={"ID":"00e44046-fb2c-481f-8221-190358d77d1a","Type":"ContainerStarted","Data":"978df90d85f49d0a65ca74dab3da2f2e4c73620704c7f3027dafa70e9a4d5575"}
Oct 31 03:36:35.099282 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:35.099222    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" podStartSLOduration=2.099206399 podStartE2EDuration="2.099206399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:35.098758877 +0000 UTC m=+1020.609741254" watchObservedRunningTime="2026-10-31 03:36:35.099206399 +0000 UTC m=+1020.610188777"
Oct 31 03:36:35.219465 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:35.219431    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:36:44.219207 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:44.219170    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:36:44.231685 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:44.231659    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:36:45.058546 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:36:45.058517    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:36:49.004164 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:36:49.004080    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:37:09.156002 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.155965    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"]
Oct 31 03:37:09.160606 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.160583    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.213136 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.213107    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-096812-openshift-default-dockercfg-7vgck\""
Oct 31 03:37:09.267205 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.267174    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"]
Oct 31 03:37:09.315436 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.315403    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrr85\" (UniqueName: \"kubernetes.io/projected/5245fec2-9af5-4816-87de-e1d9a78930b2-kube-api-access-lrr85\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.315632 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.315443    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.315632 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.315463    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.315760 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.315632    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.315760 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.315694    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.315760 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.315725    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.315940 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.315818    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.315940 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.315927    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5245fec2-9af5-4816-87de-e1d9a78930b2-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.316015 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.315991    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.417160 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.417066    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.417160 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.417117    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.417160 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.417139    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.417393 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.417166    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.417393 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.417204    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5245fec2-9af5-4816-87de-e1d9a78930b2-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.417393 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.417275    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.417393 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.417360    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrr85\" (UniqueName: \"kubernetes.io/projected/5245fec2-9af5-4816-87de-e1d9a78930b2-kube-api-access-lrr85\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.417393 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.417387    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.417651 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.417415    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.417651 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.417566    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.417790 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.417762    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.417906 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.417856    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.418151 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.417966    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.418151 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.418078    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5245fec2-9af5-4816-87de-e1d9a78930b2-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.419599 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.419578    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.419992 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.419975    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.479649 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.479618    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrr85\" (UniqueName: \"kubernetes.io/projected/5245fec2-9af5-4816-87de-e1d9a78930b2-kube-api-access-lrr85\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.479834 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.479629    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:09.775307 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:09.775273    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:10.212065 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:10.212034    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"]
Oct 31 03:37:10.212771 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:37:10.212747    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5245fec2_9af5_4816_87de_e1d9a78930b2.slice/crio-a3207a37b0b28c32d6395f541e3e411aa041e19ec8fcd7491f3e7366732954cd WatchSource:0}: Error finding container a3207a37b0b28c32d6395f541e3e411aa041e19ec8fcd7491f3e7366732954cd: Status 404 returned error can't find the container with id a3207a37b0b28c32d6395f541e3e411aa041e19ec8fcd7491f3e7366732954cd
Oct 31 03:37:10.215109 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:10.215081    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:37:10.215183 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:10.215141    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:37:10.215183 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:10.215167    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:37:10.241153 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:10.241121    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6" event={"ID":"5245fec2-9af5-4816-87de-e1d9a78930b2","Type":"ContainerStarted","Data":"a3207a37b0b28c32d6395f541e3e411aa041e19ec8fcd7491f3e7366732954cd"}
Oct 31 03:37:11.250524 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:11.250489    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6" event={"ID":"5245fec2-9af5-4816-87de-e1d9a78930b2","Type":"ContainerStarted","Data":"c07341a7a867c0bd3a6c0c4b635124ff8c096a1acc4f451e20ce43bf34b1feea"}
Oct 31 03:37:11.418314 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:11.418252    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6" podStartSLOduration=2.418232922 podStartE2EDuration="2.418232922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:11.417149057 +0000 UTC m=+1056.928131445" watchObservedRunningTime="2026-10-31 03:37:11.418232922 +0000 UTC m=+1056.929215299"
Oct 31 03:37:11.799404 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:11.797404    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:11.799404 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:11.799004    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 03:37:11.799404 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:11.799057    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6" podUID="5245fec2-9af5-4816-87de-e1d9a78930b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 03:37:12.792922 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:12.792340    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 03:37:12.792922 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:12.792407    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6" podUID="5245fec2-9af5-4816-87de-e1d9a78930b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 03:37:13.794913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:13.791369    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 03:37:13.794913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:13.791433    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6" podUID="5245fec2-9af5-4816-87de-e1d9a78930b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 03:37:14.779911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:14.778336    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 03:37:14.779911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:14.778404    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6" podUID="5245fec2-9af5-4816-87de-e1d9a78930b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 03:37:15.788927 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:15.788362    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 03:37:15.788927 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:15.788430    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6" podUID="5245fec2-9af5-4816-87de-e1d9a78930b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 03:37:16.776162 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:16.776126    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 31 03:37:16.776336 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:16.776194    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6" podUID="5245fec2-9af5-4816-87de-e1d9a78930b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 31 03:37:17.780410 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:17.780375    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:17.780822 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:17.780803    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:17.781577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:17.781543    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:37:19.003876 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:37:19.003821    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:37:23.927278 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:23.927241    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"]
Oct 31 03:37:23.931715 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:23.931695    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:23.977254 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:23.977221    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"]
Oct 31 03:37:23.981467 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:23.981444    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.025202 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.025166    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.025387 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.025233    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.025387 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.025297    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.025387 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.025360    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.025491 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.025390    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.025491 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.025405    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda8a9ea-d20f-4615-911a-46b42d93e700-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.025491 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.025441    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.025491 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.025473    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29f09a6c-204d-410b-b765-65368fd42c9d-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.025615 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.025528    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.025615 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.025593    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.025676 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.025637    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqphh\" (UniqueName: \"kubernetes.io/projected/bda8a9ea-d20f-4615-911a-46b42d93e700-kube-api-access-rqphh\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.025676 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.025658    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29f09a6c-204d-410b-b765-65368fd42c9d-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.025741 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.025708    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.025741 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.025727    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.025802 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.025766    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mphr\" (UniqueName: \"kubernetes.io/projected/29f09a6c-204d-410b-b765-65368fd42c9d-kube-api-access-6mphr\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.025802 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.025790    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.025865 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.025806    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.025865 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.025836    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29f09a6c-204d-410b-b765-65368fd42c9d-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.035392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.035363    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"]
Oct 31 03:37:24.094647 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.094154    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"]
Oct 31 03:37:24.127348 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.127312    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.127555 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.127362    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.127555 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.127408    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda8a9ea-d20f-4615-911a-46b42d93e700-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.127555 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.127453    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.127555 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.127477    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29f09a6c-204d-410b-b765-65368fd42c9d-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.127555 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.127504    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.127824 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.127668    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.127824 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.127755    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqphh\" (UniqueName: \"kubernetes.io/projected/bda8a9ea-d20f-4615-911a-46b42d93e700-kube-api-access-rqphh\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.127824 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.127790    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29f09a6c-204d-410b-b765-65368fd42c9d-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.128049 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.127825    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.128049 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.127830    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.128049 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.128025    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.128275 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.128065    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mphr\" (UniqueName: \"kubernetes.io/projected/29f09a6c-204d-410b-b765-65368fd42c9d-kube-api-access-6mphr\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.128275 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.128097    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.128275 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.128108    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.128275 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.128116    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.128275 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.128122    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.128275 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.128148    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29f09a6c-204d-410b-b765-65368fd42c9d-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.128275 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.128255    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.128757 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.128301    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda8a9ea-d20f-4615-911a-46b42d93e700-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.128757 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.128317    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-workload-certs\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.128757 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.128339    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.128757 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.128367    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.128757 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.128392    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-workload-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.128757 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.128572    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29f09a6c-204d-410b-b765-65368fd42c9d-istiod-ca-cert\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.128757 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.128576    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.129062 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.128818    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-credential-socket\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.129062 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.128818    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-istio-data\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.130814 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.130789    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.130953 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.130796    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29f09a6c-204d-410b-b765-65368fd42c9d-istio-podinfo\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.131050 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.131032    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.131124 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.131106    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-envoy\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.193485 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.193386    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29f09a6c-204d-410b-b765-65368fd42c9d-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.193636 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.193562    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqphh\" (UniqueName: \"kubernetes.io/projected/bda8a9ea-d20f-4615-911a-46b42d93e700-kube-api-access-rqphh\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.195821 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.195783    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-token\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.195821 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.195803    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mphr\" (UniqueName: \"kubernetes.io/projected/29f09a6c-204d-410b-b765-65368fd42c9d-kube-api-access-6mphr\") pod \"e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") " pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.246806 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.246778    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:24.295916 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.295851    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:24.682253 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.682221    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"]
Oct 31 03:37:24.682519 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:37:24.682499    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda8a9ea_d20f_4615_911a_46b42d93e700.slice/crio-2d336db7975013ee5cbf648ec644e79afe0179d1f1cb71d7b14ada0bdadb1b71 WatchSource:0}: Error finding container 2d336db7975013ee5cbf648ec644e79afe0179d1f1cb71d7b14ada0bdadb1b71: Status 404 returned error can't find the container with id 2d336db7975013ee5cbf648ec644e79afe0179d1f1cb71d7b14ada0bdadb1b71
Oct 31 03:37:24.685138 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.685109    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:37:24.685201 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.685171    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:37:24.685234 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.685200    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:37:24.735098 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.735073    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"]
Oct 31 03:37:24.739349 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:37:24.739321    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29f09a6c_204d_410b_b765_65368fd42c9d.slice/crio-7685258bf38b88958369462d25c788744d286cc03ebbaf8af40a278dd6c981e0 WatchSource:0}: Error finding container 7685258bf38b88958369462d25c788744d286cc03ebbaf8af40a278dd6c981e0: Status 404 returned error can't find the container with id 7685258bf38b88958369462d25c788744d286cc03ebbaf8af40a278dd6c981e0
Oct 31 03:37:24.742416 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.742382    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:37:24.742525 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.742468    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:37:24.742525 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:24.742514    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:37:25.193954 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.193917    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"]
Oct 31 03:37:25.205262 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.205230    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.250973 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.248155    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmfgx\" (UniqueName: \"kubernetes.io/projected/ce75e7a8-fe92-4059-8446-bffd5132c4b2-kube-api-access-tmfgx\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.250973 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.248292    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.250973 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.248324    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.250973 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.248356    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.250973 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.248409    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.250973 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.248436    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.250973 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.248571    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.250973 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.248595    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.250973 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.248637    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.263155 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.263107    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"]
Oct 31 03:37:25.350148 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.350112    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.350148 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.350152    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.350429 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.350188    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.350429 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.350240    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmfgx\" (UniqueName: \"kubernetes.io/projected/ce75e7a8-fe92-4059-8446-bffd5132c4b2-kube-api-access-tmfgx\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.350429 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.350350    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.350429 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.350376    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.350429 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.350404    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.350775 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.350456    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.350775 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.350483    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.350775 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.350551    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.350775 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.350577    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.351044 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.350798    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.351044 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.350874    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.351358 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.351340    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.352503 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.352486    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.353199 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.353183    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.423921 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.421822    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmfgx\" (UniqueName: \"kubernetes.io/projected/ce75e7a8-fe92-4059-8446-bffd5132c4b2-kube-api-access-tmfgx\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.423921 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.423099    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm" event={"ID":"bda8a9ea-d20f-4615-911a-46b42d93e700","Type":"ContainerStarted","Data":"3ee64b26b03331e6734c40aeccbba6e4de10aafcfb1df6f9d917b72c4e45416d"}
Oct 31 03:37:25.423921 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.423196    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm" event={"ID":"bda8a9ea-d20f-4615-911a-46b42d93e700","Type":"ContainerStarted","Data":"2d336db7975013ee5cbf648ec644e79afe0179d1f1cb71d7b14ada0bdadb1b71"}
Oct 31 03:37:25.423921 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.423538    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.428909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.425749    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt" event={"ID":"29f09a6c-204d-410b-b765-65368fd42c9d","Type":"ContainerStarted","Data":"126c377ab7e035e92301e97bf83d38983e2d93d1ace3e133925ce0a5852dbc44"}
Oct 31 03:37:25.428909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.425803    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt" event={"ID":"29f09a6c-204d-410b-b765-65368fd42c9d","Type":"ContainerStarted","Data":"7685258bf38b88958369462d25c788744d286cc03ebbaf8af40a278dd6c981e0"}
Oct 31 03:37:25.531590 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.531555    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:25.582685 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.582392    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm" podStartSLOduration=2.582370601 podStartE2EDuration="2.582370601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:25.571284053 +0000 UTC m=+1071.082266429" watchObservedRunningTime="2026-10-31 03:37:25.582370601 +0000 UTC m=+1071.093352978"
Oct 31 03:37:25.728671 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.728501    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt" podStartSLOduration=2.728478523 podStartE2EDuration="2.728478523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:25.694609338 +0000 UTC m=+1071.205591718" watchObservedRunningTime="2026-10-31 03:37:25.728478523 +0000 UTC m=+1071.239460902"
Oct 31 03:37:25.962010 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.961984    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"]
Oct 31 03:37:25.962465 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:37:25.962443    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce75e7a8_fe92_4059_8446_bffd5132c4b2.slice/crio-1b1e3f511759224d7ff1f4729caaa01d683a056097e6e197ee35c0b2ff90db1f WatchSource:0}: Error finding container 1b1e3f511759224d7ff1f4729caaa01d683a056097e6e197ee35c0b2ff90db1f: Status 404 returned error can't find the container with id 1b1e3f511759224d7ff1f4729caaa01d683a056097e6e197ee35c0b2ff90db1f
Oct 31 03:37:25.965257 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.965224    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:37:25.965349 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.965294    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:37:25.965349 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:25.965320    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:37:26.248002 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:26.247796    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:26.249339 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:26.249307    4447 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 03:37:26.249460 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:26.249370    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm" podUID="bda8a9ea-d20f-4615-911a-46b42d93e700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 03:37:26.308195 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:26.308109    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:26.308593 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:26.308485    4447 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 03:37:26.308593 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:26.308535    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt" podUID="29f09a6c-204d-410b-b765-65368fd42c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 03:37:26.441552 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:26.440633    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb" event={"ID":"ce75e7a8-fe92-4059-8446-bffd5132c4b2","Type":"ContainerStarted","Data":"1836491a2f15993b44a3abebfdb979f398ee4d2f1aa4bee9f1145d1b98ff5d8c"}
Oct 31 03:37:26.441552 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:26.441337    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb" event={"ID":"ce75e7a8-fe92-4059-8446-bffd5132c4b2","Type":"ContainerStarted","Data":"1b1e3f511759224d7ff1f4729caaa01d683a056097e6e197ee35c0b2ff90db1f"}
Oct 31 03:37:26.533070 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:26.532965    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:26.572357 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:26.572304    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb" podStartSLOduration=1.572285597 podStartE2EDuration="1.572285597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:26.571467187 +0000 UTC m=+1072.082449556" watchObservedRunningTime="2026-10-31 03:37:26.572285597 +0000 UTC m=+1072.083267977"
Oct 31 03:37:27.249598 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:27.249562    4447 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 03:37:27.250244 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:27.250200    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm" podUID="bda8a9ea-d20f-4615-911a-46b42d93e700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 03:37:27.320112 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:27.320077    4447 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 03:37:27.320445 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:27.320395    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt" podUID="29f09a6c-204d-410b-b765-65368fd42c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 03:37:27.540969 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:27.539997    4447 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 03:37:27.540969 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:27.540058    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb" podUID="ce75e7a8-fe92-4059-8446-bffd5132c4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 03:37:28.247909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:28.247309    4447 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 03:37:28.247909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:28.247372    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm" podUID="bda8a9ea-d20f-4615-911a-46b42d93e700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 03:37:28.304431 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:28.304396    4447 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 03:37:28.306347 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:28.306316    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt" podUID="29f09a6c-204d-410b-b765-65368fd42c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 03:37:28.535546 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:28.535451    4447 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 03:37:28.535808 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:28.535779    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb" podUID="ce75e7a8-fe92-4059-8446-bffd5132c4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 03:37:29.247272 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:29.247231    4447 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 03:37:29.247543 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:29.247518    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm" podUID="bda8a9ea-d20f-4615-911a-46b42d93e700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 03:37:29.300364 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:29.300326    4447 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 03:37:29.300651 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:29.300619    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt" podUID="29f09a6c-204d-410b-b765-65368fd42c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 03:37:29.532028 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:29.531931    4447 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 03:37:29.532605 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:29.532575    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb" podUID="ce75e7a8-fe92-4059-8446-bffd5132c4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 03:37:30.248912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:30.248226    4447 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 03:37:30.248912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:30.248288    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm" podUID="bda8a9ea-d20f-4615-911a-46b42d93e700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 03:37:30.304913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:30.298294    4447 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 03:37:30.304913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:30.298356    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt" podUID="29f09a6c-204d-410b-b765-65368fd42c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 03:37:30.549397 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:30.547309    4447 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 03:37:30.549397 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:30.547370    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb" podUID="ce75e7a8-fe92-4059-8446-bffd5132c4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 03:37:31.247257 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:31.247220    4447 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 03:37:31.247451 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:31.247288    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm" podUID="bda8a9ea-d20f-4615-911a-46b42d93e700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 03:37:31.300309 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:31.300266    4447 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 03:37:31.300601 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:31.300568    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt" podUID="29f09a6c-204d-410b-b765-65368fd42c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 03:37:31.533083 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:31.532961    4447 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 03:37:31.533083 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:31.533045    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb" podUID="ce75e7a8-fe92-4059-8446-bffd5132c4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 03:37:32.252759 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:32.252726    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:32.253222 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:32.253203    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:32.254096 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:32.254077    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:37:32.302719 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:32.302693    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:32.303039 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:32.303017    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:32.303801 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:32.303726    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:37:32.532530 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:32.532440    4447 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 03:37:32.532530 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:32.532515    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb" podUID="ce75e7a8-fe92-4059-8446-bffd5132c4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 03:37:33.536783 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:33.536750    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:33.537254 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:33.537233    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:33.537975 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:37:33.537957    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:37:49.025028 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:37:49.024981    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:38:19.006962 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:38:19.006873    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:38:49.004495 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:38:49.004460    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:39:03.002336 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.002302    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"]
Oct 31 03:39:03.006978 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.006952    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:39:03.066350 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.066319    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"]
Oct 31 03:39:03.099688 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.099652    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-tmp-dir\") pod \"distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z\" (UID: \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:39:03.099908 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.099784    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-dshm\") pod \"distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z\" (UID: \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:39:03.099908 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.099823    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8d2t\" (UniqueName: \"kubernetes.io/projected/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-kube-api-access-d8d2t\") pod \"distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z\" (UID: \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:39:03.100006 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.099942    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-tls-certs\") pod \"distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z\" (UID: \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:39:03.100070 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.100056    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-home\") pod \"distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z\" (UID: \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:39:03.100130 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.100116    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-model-cache\") pod \"distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z\" (UID: \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:39:03.111262 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.111235    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e00ad3\"/\"openshift-service-ca.crt\""
Oct 31 03:39:03.111412 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.111269    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e00ad3\"/\"distinct-2-w1-e00ad3-kserve-self-signed-certs\""
Oct 31 03:39:03.111412 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.111283    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e00ad3\"/\"default-dockercfg-lsfwl\""
Oct 31 03:39:03.111490 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.111417    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e00ad3\"/\"kube-root-ca.crt\""
Oct 31 03:39:03.201401 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.201359    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-tmp-dir\") pod \"distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z\" (UID: \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:39:03.201581 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.201454    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-dshm\") pod \"distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z\" (UID: \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:39:03.201581 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.201472    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8d2t\" (UniqueName: \"kubernetes.io/projected/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-kube-api-access-d8d2t\") pod \"distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z\" (UID: \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:39:03.201581 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.201503    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-tls-certs\") pod \"distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z\" (UID: \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:39:03.201746 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.201618    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-home\") pod \"distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z\" (UID: \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:39:03.201746 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.201654    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-model-cache\") pod \"distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z\" (UID: \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:39:03.201948 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.201878    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-tmp-dir\") pod \"distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z\" (UID: \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:39:03.202207 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.202127    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-home\") pod \"distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z\" (UID: \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:39:03.202207 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.202189    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-model-cache\") pod \"distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z\" (UID: \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:39:03.204289 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.204264    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-dshm\") pod \"distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z\" (UID: \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:39:03.204602 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.204584    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-tls-certs\") pod \"distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z\" (UID: \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:39:03.263665 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.263586    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8d2t\" (UniqueName: \"kubernetes.io/projected/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-kube-api-access-d8d2t\") pod \"distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z\" (UID: \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\") " pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:39:03.319974 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.319943    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:39:03.713310 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:03.713283    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"]
Oct 31 03:39:04.227350 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:04.227313    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" event={"ID":"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de","Type":"ContainerStarted","Data":"8b49f5b4b107f4eaa3cc38e556fdd7a376ae95efc408c134f56bcee71c77a7f0"}
Oct 31 03:39:04.227749 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:04.227373    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" event={"ID":"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de","Type":"ContainerStarted","Data":"1fbc89af967802bf6149f1f72907a10855ef1b2ef8e982af8d046d2c2d95714b"}
Oct 31 03:39:04.320074 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:04.320040    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:39:04.349738 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:04.349686    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" podStartSLOduration=2.34967054 podStartE2EDuration="2.34967054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:04.345636143 +0000 UTC m=+1169.856618519" watchObservedRunningTime="2026-10-31 03:39:04.34967054 +0000 UTC m=+1169.860652928"
Oct 31 03:39:13.320502 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:13.320466    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:39:13.333459 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:13.333425    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:39:14.322311 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:14.322278    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:39:19.006044 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:39:19.006005    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:39:20.525493 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.525459    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"]
Oct 31 03:39:20.529954 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.529929    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:39:20.633594 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.633559    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-b63425\"/\"kube-root-ca.crt\""
Oct 31 03:39:20.633783 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.633611    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-b63425\"/\"openshift-service-ca.crt\""
Oct 31 03:39:20.633783 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.633651    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-b63425\"/\"distinct-2-w6-b63425-kserve-self-signed-certs\""
Oct 31 03:39:20.633783 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.633652    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-b63425\"/\"default-dockercfg-5rnfw\""
Oct 31 03:39:20.637738 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.637715    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"]
Oct 31 03:39:20.642264 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.642246    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-tmp-dir\") pod \"distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq\" (UID: \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:39:20.642343 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.642300    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b8712d5-205e-4fc4-bc62-9b35dca0310a-tls-certs\") pod \"distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq\" (UID: \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:39:20.642388 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.642364    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-home\") pod \"distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq\" (UID: \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:39:20.642430 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.642420    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-model-cache\") pod \"distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq\" (UID: \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:39:20.642480 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.642457    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch77c\" (UniqueName: \"kubernetes.io/projected/6b8712d5-205e-4fc4-bc62-9b35dca0310a-kube-api-access-ch77c\") pod \"distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq\" (UID: \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:39:20.642557 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.642541    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-dshm\") pod \"distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq\" (UID: \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:39:20.743804 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.743763    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-model-cache\") pod \"distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq\" (UID: \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:39:20.744058 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.743830    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch77c\" (UniqueName: \"kubernetes.io/projected/6b8712d5-205e-4fc4-bc62-9b35dca0310a-kube-api-access-ch77c\") pod \"distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq\" (UID: \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:39:20.744058 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.743961    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-dshm\") pod \"distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq\" (UID: \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:39:20.744202 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.744143    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-tmp-dir\") pod \"distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq\" (UID: \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:39:20.744202 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.744196    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b8712d5-205e-4fc4-bc62-9b35dca0310a-tls-certs\") pod \"distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq\" (UID: \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:39:20.744400 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.744220    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-model-cache\") pod \"distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq\" (UID: \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:39:20.744400 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.744229    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-home\") pod \"distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq\" (UID: \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:39:20.744512 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.744491    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-tmp-dir\") pod \"distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq\" (UID: \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:39:20.744552 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.744533    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-home\") pod \"distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq\" (UID: \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:39:20.746336 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.746310    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-dshm\") pod \"distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq\" (UID: \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:39:20.746760 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.746744    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b8712d5-205e-4fc4-bc62-9b35dca0310a-tls-certs\") pod \"distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq\" (UID: \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:39:20.805400 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.805330    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch77c\" (UniqueName: \"kubernetes.io/projected/6b8712d5-205e-4fc4-bc62-9b35dca0310a-kube-api-access-ch77c\") pod \"distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq\" (UID: \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\") " pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:39:20.845131 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:20.845087    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:39:21.245607 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:21.245575    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"]
Oct 31 03:39:21.246965 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:39:21.246935    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b8712d5_205e_4fc4_bc62_9b35dca0310a.slice/crio-ee5999556a3007eaf78aa9614a5c13d1d567eaa94d0371bfdea464c7065cc9e8 WatchSource:0}: Error finding container ee5999556a3007eaf78aa9614a5c13d1d567eaa94d0371bfdea464c7065cc9e8: Status 404 returned error can't find the container with id ee5999556a3007eaf78aa9614a5c13d1d567eaa94d0371bfdea464c7065cc9e8
Oct 31 03:39:21.363587 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:21.363542    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" event={"ID":"6b8712d5-205e-4fc4-bc62-9b35dca0310a","Type":"ContainerStarted","Data":"7f28dc5c84cfcd5917b9ab87beb931beb5a67452082a67eca1650b8e8a028f01"}
Oct 31 03:39:21.363777 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:21.363596    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" event={"ID":"6b8712d5-205e-4fc4-bc62-9b35dca0310a","Type":"ContainerStarted","Data":"ee5999556a3007eaf78aa9614a5c13d1d567eaa94d0371bfdea464c7065cc9e8"}
Oct 31 03:39:21.487394 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:21.487341    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" podStartSLOduration=1.487327731 podStartE2EDuration="1.487327731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:21.481477049 +0000 UTC m=+1186.992459427" watchObservedRunningTime="2026-10-31 03:39:21.487327731 +0000 UTC m=+1186.998310108"
Oct 31 03:39:21.845336 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:21.845304    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:39:25.868983 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:25.868951    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"]
Oct 31 03:39:25.873307 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:25.873291    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:25.914607 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:25.914574    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:25.914766 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:25.914627    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:25.914766 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:25.914709    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:25.914766 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:25.914759    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:25.914873 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:25.914855    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:25.914947 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:25.914879    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:25.915019 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:25.915006    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:25.915053 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:25.915027    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7cjw\" (UniqueName: \"kubernetes.io/projected/5f030d4e-f850-4a06-a262-d2060e084bbf-kube-api-access-d7cjw\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:25.915088 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:25.915052    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f030d4e-f850-4a06-a262-d2060e084bbf-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:26.016853 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:26.016814    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:26.017092 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:26.016865    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:26.017092 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:26.016944    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:26.017092 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:26.017021    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:26.017092 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:26.017044    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:26.017382 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:26.017135    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:26.017382 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:26.017161    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7cjw\" (UniqueName: \"kubernetes.io/projected/5f030d4e-f850-4a06-a262-d2060e084bbf-kube-api-access-d7cjw\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:26.017382 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:26.017200    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f030d4e-f850-4a06-a262-d2060e084bbf-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:26.017382 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:26.017324    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:26.017597 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:26.017396    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:26.017597 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:26.017429    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:26.017597 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:26.017473    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:26.017597 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:26.017511    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:26.018025 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:26.018003    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f030d4e-f850-4a06-a262-d2060e084bbf-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:26.019518 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:26.019495    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:26.019518 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:26.019516    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:26.078191 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:26.078155    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:26.078334 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:26.078261    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7cjw\" (UniqueName: \"kubernetes.io/projected/5f030d4e-f850-4a06-a262-d2060e084bbf-kube-api-access-d7cjw\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:26.087373 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:26.087346    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"]
Oct 31 03:39:26.188299 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:26.188204    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:26.635484 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:39:26.635451    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f030d4e_f850_4a06_a262_d2060e084bbf.slice/crio-2735a1eaba9bf040d6169f53579a7dbf85636ba1a41fe4791d3a25e8918daf45 WatchSource:0}: Error finding container 2735a1eaba9bf040d6169f53579a7dbf85636ba1a41fe4791d3a25e8918daf45: Status 404 returned error can't find the container with id 2735a1eaba9bf040d6169f53579a7dbf85636ba1a41fe4791d3a25e8918daf45
Oct 31 03:39:26.638327 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:26.638265    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:39:26.638444 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:26.638381    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:39:26.638444 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:26.638422    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:39:26.638663 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:26.638634    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"]
Oct 31 03:39:27.459907 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:27.459437    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v" event={"ID":"5f030d4e-f850-4a06-a262-d2060e084bbf","Type":"ContainerStarted","Data":"f97383e94f5f715c98d147432d45f39b4d49bb03c2efe9896601d6999fcaf65e"}
Oct 31 03:39:27.459907 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:27.459511    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v" event={"ID":"5f030d4e-f850-4a06-a262-d2060e084bbf","Type":"ContainerStarted","Data":"2735a1eaba9bf040d6169f53579a7dbf85636ba1a41fe4791d3a25e8918daf45"}
Oct 31 03:39:27.653911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:27.653232    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v" podStartSLOduration=2.653210148 podStartE2EDuration="2.653210148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:27.650349546 +0000 UTC m=+1193.161331933" watchObservedRunningTime="2026-10-31 03:39:27.653210148 +0000 UTC m=+1193.164192526"
Oct 31 03:39:28.303910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:28.292554    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:28.326910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:28.324067    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 03:39:28.326910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:28.324148    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v" podUID="5f030d4e-f850-4a06-a262-d2060e084bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 03:39:29.190910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:29.189307    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 03:39:29.190910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:29.189367    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v" podUID="5f030d4e-f850-4a06-a262-d2060e084bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 03:39:30.204910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:30.202055    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 03:39:30.204910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:30.202117    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v" podUID="5f030d4e-f850-4a06-a262-d2060e084bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 03:39:30.845293 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:30.845265    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:39:30.857344 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:30.857321    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:39:31.194476 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:31.194382    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 03:39:31.194722 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:31.194690    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v" podUID="5f030d4e-f850-4a06-a262-d2060e084bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 03:39:31.549761 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:31.549735    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:39:32.199122 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:32.196618    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 03:39:32.199122 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:32.196698    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v" podUID="5f030d4e-f850-4a06-a262-d2060e084bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 03:39:33.189475 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:33.189430    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 03:39:33.190017 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:33.189507    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v" podUID="5f030d4e-f850-4a06-a262-d2060e084bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 03:39:34.517772 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:34.517740    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:34.518664 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:34.518636    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:34.519683 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:34.519625    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:39:34.790676 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:34.790597    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 03:39:34.802082 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:34.802024    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 03:39:41.249970 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.249935    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"]
Oct 31 03:39:41.256491 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.256468    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.339062 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.339022    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"]
Oct 31 03:39:41.389324 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.389292    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab02233-a047-4c35-9366-573b2dfd10cb-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.389491 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.389330    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab02233-a047-4c35-9366-573b2dfd10cb-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.389491 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.389448    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.389558 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.389513    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eab02233-a047-4c35-9366-573b2dfd10cb-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.389558 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.389548    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.389634 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.389616    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wzbx\" (UniqueName: \"kubernetes.io/projected/eab02233-a047-4c35-9366-573b2dfd10cb-kube-api-access-8wzbx\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.389749 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.389732    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.389784 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.389761    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.389818 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.389781    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.491643 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.491605    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.491814 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.491712    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eab02233-a047-4c35-9366-573b2dfd10cb-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.491814 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.491753    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.491814 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.491794    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wzbx\" (UniqueName: \"kubernetes.io/projected/eab02233-a047-4c35-9366-573b2dfd10cb-kube-api-access-8wzbx\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.491986 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.491871    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.491986 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.491920    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.492090 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.492025    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.492212 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.492188    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-workload-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.492288 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.492198    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab02233-a047-4c35-9366-573b2dfd10cb-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.492288 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.492260    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab02233-a047-4c35-9366-573b2dfd10cb-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.492395 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.492292    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-workload-certs\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.492583 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.492561    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-istio-data\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.492814 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.492784    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-credential-socket\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.492910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.492842    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab02233-a047-4c35-9366-573b2dfd10cb-istiod-ca-cert\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.494216 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.494198    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-istio-envoy\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.494757 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.494741    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab02233-a047-4c35-9366-573b2dfd10cb-istio-podinfo\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.553735 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.553661    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eab02233-a047-4c35-9366-573b2dfd10cb-istio-token\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.553735 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.553728    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wzbx\" (UniqueName: \"kubernetes.io/projected/eab02233-a047-4c35-9366-573b2dfd10cb-kube-api-access-8wzbx\") pod \"e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") " pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:41.571079 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:41.571050    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:42.012856 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:42.012824    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"]
Oct 31 03:39:42.013000 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:39:42.012863    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab02233_a047_4c35_9366_573b2dfd10cb.slice/crio-ea6e7913a1f7f6ef2e9450dc40b3f1a3ebbe94eb9213a893d0d17ac4ec872d2a WatchSource:0}: Error finding container ea6e7913a1f7f6ef2e9450dc40b3f1a3ebbe94eb9213a893d0d17ac4ec872d2a: Status 404 returned error can't find the container with id ea6e7913a1f7f6ef2e9450dc40b3f1a3ebbe94eb9213a893d0d17ac4ec872d2a
Oct 31 03:39:42.015199 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:42.015164    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:39:42.015310 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:42.015259    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:39:42.015310 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:42.015303    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:39:42.629078 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:42.629043    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch" event={"ID":"eab02233-a047-4c35-9366-573b2dfd10cb","Type":"ContainerStarted","Data":"b8f35cb7dff26b752da6d37e184e5de01905f4b60fd566649d97363276b12584"}
Oct 31 03:39:42.629514 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:42.629098    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch" event={"ID":"eab02233-a047-4c35-9366-573b2dfd10cb","Type":"ContainerStarted","Data":"ea6e7913a1f7f6ef2e9450dc40b3f1a3ebbe94eb9213a893d0d17ac4ec872d2a"}
Oct 31 03:39:42.762787 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:42.762725    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch" podStartSLOduration=1.7627053780000002 podStartE2EDuration="1.762705378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:42.762048643 +0000 UTC m=+1208.273031048" watchObservedRunningTime="2026-10-31 03:39:42.762705378 +0000 UTC m=+1208.273687757"
Oct 31 03:39:43.571548 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:43.571518    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:43.573008 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:43.572984    4447 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 03:39:43.573147 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:43.573035    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch" podUID="eab02233-a047-4c35-9366-573b2dfd10cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 03:39:44.571543 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:44.571512    4447 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 03:39:44.572093 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:44.571559    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch" podUID="eab02233-a047-4c35-9366-573b2dfd10cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 03:39:45.131137 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:45.131105    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"]
Oct 31 03:39:45.131340 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:45.131307    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng" podUID="7c79e465-b30e-482a-bccb-61f5bcb4d910" containerName="istio-proxy" containerID="cri-o://51f52ddfd8b706a51037a519f91d59a73143ec4b41a2c12a6b8781e015a53f23" gracePeriod=30
Oct 31 03:39:45.571817 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:45.571775    4447 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 03:39:45.572273 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:45.571849    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch" podUID="eab02233-a047-4c35-9366-573b2dfd10cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 03:39:46.299821 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:46.299727    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-9qtng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:39:46.299821 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:46.299793    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng" podUID="7c79e465-b30e-482a-bccb-61f5bcb4d910" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:39:46.572169 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:46.572079    4447 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 03:39:46.572169 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:46.572131    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch" podUID="eab02233-a047-4c35-9366-573b2dfd10cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 03:39:47.572177 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:47.572142    4447 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 03:39:47.572592 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:47.572199    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch" podUID="eab02233-a047-4c35-9366-573b2dfd10cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 03:39:48.571665 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:48.571630    4447 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 03:39:48.571869 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:48.571688    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch" podUID="eab02233-a047-4c35-9366-573b2dfd10cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 03:39:49.004867 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:39:49.004832    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:39:49.576273 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:49.576238    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:49.576611 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:49.576592    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:49.577337 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:49.577317    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 03:39:50.413871 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.413807    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:39:50.541485 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.541452    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-credential-socket\") pod \"7c79e465-b30e-482a-bccb-61f5bcb4d910\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") "
Oct 31 03:39:50.541675 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.541514    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c79e465-b30e-482a-bccb-61f5bcb4d910-kube-api-access-8mn84\" (UniqueName: \"kubernetes.io/projected/7c79e465-b30e-482a-bccb-61f5bcb4d910-kube-api-access-8mn84\") pod \"7c79e465-b30e-482a-bccb-61f5bcb4d910\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") "
Oct 31 03:39:50.541675 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.541572    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-envoy\") pod \"7c79e465-b30e-482a-bccb-61f5bcb4d910\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") "
Oct 31 03:39:50.541675 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.541600    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-data\") pod \"7c79e465-b30e-482a-bccb-61f5bcb4d910\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") "
Oct 31 03:39:50.541675 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.541645    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-podinfo\") pod \"7c79e465-b30e-482a-bccb-61f5bcb4d910\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") "
Oct 31 03:39:50.541902 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.541726    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-token\" (UniqueName: \"kubernetes.io/projected/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-token\") pod \"7c79e465-b30e-482a-bccb-61f5bcb4d910\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") "
Oct 31 03:39:50.541902 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.541774    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-credential-socket" pod "7c79e465-b30e-482a-bccb-61f5bcb4d910" (UID: "7c79e465-b30e-482a-bccb-61f5bcb4d910"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:50.541902 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.541831    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c79e465-b30e-482a-bccb-61f5bcb4d910-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c79e465-b30e-482a-bccb-61f5bcb4d910-istiod-ca-cert\") pod \"7c79e465-b30e-482a-bccb-61f5bcb4d910\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") "
Oct 31 03:39:50.542051 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.541968    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-workload-certs\") pod \"7c79e465-b30e-482a-bccb-61f5bcb4d910\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") "
Oct 31 03:39:50.542051 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.542019    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-data" pod "7c79e465-b30e-482a-bccb-61f5bcb4d910" (UID: "7c79e465-b30e-482a-bccb-61f5bcb4d910"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:50.542156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.542076    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-workload-socket\") pod \"7c79e465-b30e-482a-bccb-61f5bcb4d910\" (UID: \"7c79e465-b30e-482a-bccb-61f5bcb4d910\") "
Oct 31 03:39:50.542322 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.542299    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-workload-certs" pod "7c79e465-b30e-482a-bccb-61f5bcb4d910" (UID: "7c79e465-b30e-482a-bccb-61f5bcb4d910"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:50.542608 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.542345    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-workload-socket" pod "7c79e465-b30e-482a-bccb-61f5bcb4d910" (UID: "7c79e465-b30e-482a-bccb-61f5bcb4d910"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:50.542608 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.542451    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c79e465-b30e-482a-bccb-61f5bcb4d910-istiod-ca-cert" pod "7c79e465-b30e-482a-bccb-61f5bcb4d910" (UID: "7c79e465-b30e-482a-bccb-61f5bcb4d910"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:39:50.542854 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.542760    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c79e465-b30e-482a-bccb-61f5bcb4d910-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:39:50.542854 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.542786    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:39:50.542854 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.542802    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:39:50.542854 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.542816    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:39:50.542854 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.542830    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:39:50.544559 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.544513    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-token" pod "7c79e465-b30e-482a-bccb-61f5bcb4d910" (UID: "7c79e465-b30e-482a-bccb-61f5bcb4d910"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:50.544688 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.544626    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c79e465-b30e-482a-bccb-61f5bcb4d910-kube-api-access-8mn84" pod "7c79e465-b30e-482a-bccb-61f5bcb4d910" (UID: "7c79e465-b30e-482a-bccb-61f5bcb4d910"). InnerVolumeSpecName "kube-api-access-8mn84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:50.544852 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.544826    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-envoy" pod "7c79e465-b30e-482a-bccb-61f5bcb4d910" (UID: "7c79e465-b30e-482a-bccb-61f5bcb4d910"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:50.545051 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.545021    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-podinfo" pod "7c79e465-b30e-482a-bccb-61f5bcb4d910" (UID: "7c79e465-b30e-482a-bccb-61f5bcb4d910"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:39:50.644616 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.644577    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mn84\" (UniqueName: \"kubernetes.io/projected/7c79e465-b30e-482a-bccb-61f5bcb4d910-kube-api-access-8mn84\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:39:50.644616 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.644616    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:39:50.644816 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.644633    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:39:50.644816 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.644645    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c79e465-b30e-482a-bccb-61f5bcb4d910-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:39:50.689009 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.688862    4447 generic.go:356] "Generic (PLEG): container finished" podID="7c79e465-b30e-482a-bccb-61f5bcb4d910" containerID="51f52ddfd8b706a51037a519f91d59a73143ec4b41a2c12a6b8781e015a53f23" exitCode=0
Oct 31 03:39:50.689009 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.688957    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"
Oct 31 03:39:50.689009 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.688952    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng" event={"ID":"7c79e465-b30e-482a-bccb-61f5bcb4d910","Type":"ContainerDied","Data":"51f52ddfd8b706a51037a519f91d59a73143ec4b41a2c12a6b8781e015a53f23"}
Oct 31 03:39:50.689239 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.689018    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng" event={"ID":"7c79e465-b30e-482a-bccb-61f5bcb4d910","Type":"ContainerDied","Data":"c285028bfa50a6de1a747ec89c3654785848fbc4ecfc9c3e40056f10068b3b4b"}
Oct 31 03:39:50.689239 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.689043    4447 scope.go:122] "RemoveContainer" containerID="51f52ddfd8b706a51037a519f91d59a73143ec4b41a2c12a6b8781e015a53f23"
Oct 31 03:39:50.704037 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.704015    4447 scope.go:122] "RemoveContainer" containerID="c7653d66d44d682f4e02f0195c77262823daf9c3dc1e17068cbe803a7d626110"
Oct 31 03:39:50.714257 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.714231    4447 scope.go:122] "RemoveContainer" containerID="51f52ddfd8b706a51037a519f91d59a73143ec4b41a2c12a6b8781e015a53f23"
Oct 31 03:39:50.714588 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:39:50.714568    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51f52ddfd8b706a51037a519f91d59a73143ec4b41a2c12a6b8781e015a53f23\": container with ID starting with 51f52ddfd8b706a51037a519f91d59a73143ec4b41a2c12a6b8781e015a53f23 not found: ID does not exist" containerID="51f52ddfd8b706a51037a519f91d59a73143ec4b41a2c12a6b8781e015a53f23"
Oct 31 03:39:50.714645 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.714601    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51f52ddfd8b706a51037a519f91d59a73143ec4b41a2c12a6b8781e015a53f23"} err="failed to get container status \"51f52ddfd8b706a51037a519f91d59a73143ec4b41a2c12a6b8781e015a53f23\": rpc error: code = NotFound desc = could not find container \"51f52ddfd8b706a51037a519f91d59a73143ec4b41a2c12a6b8781e015a53f23\": container with ID starting with 51f52ddfd8b706a51037a519f91d59a73143ec4b41a2c12a6b8781e015a53f23 not found: ID does not exist"
Oct 31 03:39:50.714645 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.714618    4447 scope.go:122] "RemoveContainer" containerID="c7653d66d44d682f4e02f0195c77262823daf9c3dc1e17068cbe803a7d626110"
Oct 31 03:39:50.714869 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:39:50.714852    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7653d66d44d682f4e02f0195c77262823daf9c3dc1e17068cbe803a7d626110\": container with ID starting with c7653d66d44d682f4e02f0195c77262823daf9c3dc1e17068cbe803a7d626110 not found: ID does not exist" containerID="c7653d66d44d682f4e02f0195c77262823daf9c3dc1e17068cbe803a7d626110"
Oct 31 03:39:50.714941 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.714878    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7653d66d44d682f4e02f0195c77262823daf9c3dc1e17068cbe803a7d626110"} err="failed to get container status \"c7653d66d44d682f4e02f0195c77262823daf9c3dc1e17068cbe803a7d626110\": rpc error: code = NotFound desc = could not find container \"c7653d66d44d682f4e02f0195c77262823daf9c3dc1e17068cbe803a7d626110\": container with ID starting with c7653d66d44d682f4e02f0195c77262823daf9c3dc1e17068cbe803a7d626110 not found: ID does not exist"
Oct 31 03:39:50.857568 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.857533    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"]
Oct 31 03:39:50.861709 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:50.861685    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-9qtng"]
Oct 31 03:39:52.608892 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:39:52.608851    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c79e465-b30e-482a-bccb-61f5bcb4d910" path="/var/lib/kubelet/pods/7c79e465-b30e-482a-bccb-61f5bcb4d910/volumes"
Oct 31 03:40:19.004186 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:40:19.004150    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:40:26.361964 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.361916    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"]
Oct 31 03:40:26.362619 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.362605    4447 cpu_manager.go:414] "containerMap: removing container" podUID="7c79e465-b30e-482a-bccb-61f5bcb4d910" containerName="istio-proxy"
Oct 31 03:40:26.362665 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.362620    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c79e465-b30e-482a-bccb-61f5bcb4d910" containerName="istio-proxy"
Oct 31 03:40:26.362665 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.362655    4447 cpu_manager.go:414] "containerMap: removing container" podUID="7c79e465-b30e-482a-bccb-61f5bcb4d910" containerName="istio-proxy"
Oct 31 03:40:26.362665 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.362660    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c79e465-b30e-482a-bccb-61f5bcb4d910" containerName="istio-proxy"
Oct 31 03:40:26.362776 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.362768    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c79e465-b30e-482a-bccb-61f5bcb4d910" containerName="istio-proxy"
Oct 31 03:40:26.362811 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.362784    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c79e465-b30e-482a-bccb-61f5bcb4d910" containerName="istio-proxy"
Oct 31 03:40:26.366569 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.366551    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.375285 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.375255    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f503715-aa26-4a05-9391-de0b40a919e7-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.375410 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.375306    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82jvr\" (UniqueName: \"kubernetes.io/projected/9f503715-aa26-4a05-9391-de0b40a919e7-kube-api-access-82jvr\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.375487 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.375469    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f503715-aa26-4a05-9391-de0b40a919e7-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.375601 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.375587    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.375686 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.375642    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.375755 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.375739    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.375816 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.375794    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.375868 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.375840    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f503715-aa26-4a05-9391-de0b40a919e7-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.398257 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.375991    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.474908 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.474857    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"]
Oct 31 03:40:26.477391 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.477360    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f503715-aa26-4a05-9391-de0b40a919e7-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.478382 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.478354    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82jvr\" (UniqueName: \"kubernetes.io/projected/9f503715-aa26-4a05-9391-de0b40a919e7-kube-api-access-82jvr\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.478626 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.478608    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f503715-aa26-4a05-9391-de0b40a919e7-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.478864 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.478826    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.478864 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.478863    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.479127 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.478920    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.479127 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.478953    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.479127 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.479003    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f503715-aa26-4a05-9391-de0b40a919e7-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.479127 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.479067    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.479388 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.479364    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-workload-certs\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.479590 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.479573    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f503715-aa26-4a05-9391-de0b40a919e7-istiod-ca-cert\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.479694 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.479641    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-credential-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.479776 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.479718    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-istio-data\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.479988 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.479965    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-workload-socket\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.482505 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.482481    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-istio-envoy\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.482643 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.482621    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f503715-aa26-4a05-9391-de0b40a919e7-istio-podinfo\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.540121 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.540081    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82jvr\" (UniqueName: \"kubernetes.io/projected/9f503715-aa26-4a05-9391-de0b40a919e7-kube-api-access-82jvr\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.540467 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.540447    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f503715-aa26-4a05-9391-de0b40a919e7-istio-token\") pod \"e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") " pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:26.696134 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:26.696050    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:27.132094 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:27.132063    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"]
Oct 31 03:40:27.132341 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:40:27.132319    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f503715_aa26_4a05_9391_de0b40a919e7.slice/crio-fa3572d27a6b219eb7f51422ff2ca99de8cbb4fe31f8c43902ce180d3a3748a5 WatchSource:0}: Error finding container fa3572d27a6b219eb7f51422ff2ca99de8cbb4fe31f8c43902ce180d3a3748a5: Status 404 returned error can't find the container with id fa3572d27a6b219eb7f51422ff2ca99de8cbb4fe31f8c43902ce180d3a3748a5
Oct 31 03:40:27.134736 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:27.134702    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:40:27.134850 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:27.134779    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:40:27.134850 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:27.134822    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:40:27.965634 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:27.965601    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn" event={"ID":"9f503715-aa26-4a05-9391-de0b40a919e7","Type":"ContainerStarted","Data":"f0b290178f5a30e4f2878a124b1baae346018dac8287fcd5c025dd04dd6ff3f0"}
Oct 31 03:40:27.966166 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:27.965668    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn" event={"ID":"9f503715-aa26-4a05-9391-de0b40a919e7","Type":"ContainerStarted","Data":"fa3572d27a6b219eb7f51422ff2ca99de8cbb4fe31f8c43902ce180d3a3748a5"}
Oct 31 03:40:28.139279 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:28.139226    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn" podStartSLOduration=2.139212148 podStartE2EDuration="2.139212148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:28.136233353 +0000 UTC m=+1253.647215732" watchObservedRunningTime="2026-10-31 03:40:28.139212148 +0000 UTC m=+1253.650194524"
Oct 31 03:40:28.701182 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:28.701141    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:28.702733 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:28.702707    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 03:40:28.702876 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:28.702773    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn" podUID="9f503715-aa26-4a05-9391-de0b40a919e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 03:40:29.700401 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:29.700366    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 03:40:29.701043 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:29.700431    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn" podUID="9f503715-aa26-4a05-9391-de0b40a919e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 03:40:30.699466 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:30.699419    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 03:40:30.699675 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:30.699478    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn" podUID="9f503715-aa26-4a05-9391-de0b40a919e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 03:40:31.050607 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:31.050568    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c11ccea-sqgjs"]
Oct 31 03:40:31.104940 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:31.104905    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c11ccea-sqgjs"]
Oct 31 03:40:31.708911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:31.700999    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 03:40:31.708911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:31.701058    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn" podUID="9f503715-aa26-4a05-9391-de0b40a919e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 03:40:32.666913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:32.646698    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18ad860c-f831-4582-9f6b-700c1edf2c67" path="/var/lib/kubelet/pods/18ad860c-f831-4582-9f6b-700c1edf2c67/volumes"
Oct 31 03:40:32.696917 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:32.696405    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 03:40:32.696917 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:32.696461    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn" podUID="9f503715-aa26-4a05-9391-de0b40a919e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 03:40:33.702179 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:33.702142    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 03:40:33.702785 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:33.702753    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn" podUID="9f503715-aa26-4a05-9391-de0b40a919e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 03:40:33.852045 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:33.852007    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:40:33.852371 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:33.852327    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6" podUID="5245fec2-9af5-4816-87de-e1d9a78930b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:40:34.074918 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:34.065155    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 31 03:40:34.247048 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:34.247001    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" podUID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:40:34.308261 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:34.308217    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 31 03:40:34.328146 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:34.327989    4447 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:40:34.574918 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:34.551173    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded"
Oct 31 03:40:34.818987 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:34.814549    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 03:40:34.838371 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:34.837109    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn" podUID="9f503715-aa26-4a05-9391-de0b40a919e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 03:40:35.072974 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:35.061304    4447 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded"
Oct 31 03:40:35.072974 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:35.070067    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:40:35.605118 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:35.605075    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:40:35.637482 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:35.637444    4447 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": context deadline exceeded"
Oct 31 03:40:35.650924 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:35.649576    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" podUID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:40:35.653606 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:35.653543    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:40:35.661827 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:35.654971    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:40:35.765832 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:40:35.742413    4447 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.062s"
Oct 31 03:40:35.765832 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:35.742478    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:40:35.765832 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:35.742540    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:40:35.765832 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:35.742571    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:40:35.765832 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:35.742603    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:40:35.765832 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:35.742631    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:40:35.780629 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:35.780569    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 03:40:35.800380 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:35.800150    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn" podUID="9f503715-aa26-4a05-9391-de0b40a919e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 03:40:35.807334 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:35.807197    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:40:35.817056 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:35.817026    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:40:35.843026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:35.843000    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:40:37.552864 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:37.552832    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:37.554952 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:37.554898    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:37.555353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:37.555333    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:40:39.200815 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:39.200782    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-607bc420-bhtbg"]
Oct 31 03:40:39.206486 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:39.206466    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607bc420-bhtbg"
Oct 31 03:40:39.249271 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:39.249240    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13313afc-657b-4a56-a50d-e0547d6781de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-607bc420-bhtbg\" (UID: \"13313afc-657b-4a56-a50d-e0547d6781de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607bc420-bhtbg"
Oct 31 03:40:39.249447 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:39.249298    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13313afc-657b-4a56-a50d-e0547d6781de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-607bc420-bhtbg\" (UID: \"13313afc-657b-4a56-a50d-e0547d6781de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607bc420-bhtbg"
Oct 31 03:40:39.249496 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:39.249463    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrj67\" (UniqueName: \"kubernetes.io/projected/13313afc-657b-4a56-a50d-e0547d6781de-kube-api-access-zrj67\") pod \"maas-api-revoke-sub-607bc420-bhtbg\" (UID: \"13313afc-657b-4a56-a50d-e0547d6781de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607bc420-bhtbg"
Oct 31 03:40:39.263039 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:39.263010    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-607bc420-bhtbg"]
Oct 31 03:40:39.307650 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:39.307621    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:40:39.308044 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:39.307697    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 03:40:39.351030 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:39.350998    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13313afc-657b-4a56-a50d-e0547d6781de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-607bc420-bhtbg\" (UID: \"13313afc-657b-4a56-a50d-e0547d6781de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607bc420-bhtbg"
Oct 31 03:40:39.351206 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:39.351089    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrj67\" (UniqueName: \"kubernetes.io/projected/13313afc-657b-4a56-a50d-e0547d6781de-kube-api-access-zrj67\") pod \"maas-api-revoke-sub-607bc420-bhtbg\" (UID: \"13313afc-657b-4a56-a50d-e0547d6781de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607bc420-bhtbg"
Oct 31 03:40:39.351342 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:39.351324    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13313afc-657b-4a56-a50d-e0547d6781de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-607bc420-bhtbg\" (UID: \"13313afc-657b-4a56-a50d-e0547d6781de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607bc420-bhtbg"
Oct 31 03:40:39.352012 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:39.351993    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13313afc-657b-4a56-a50d-e0547d6781de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-607bc420-bhtbg\" (UID: \"13313afc-657b-4a56-a50d-e0547d6781de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607bc420-bhtbg"
Oct 31 03:40:39.353644 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:39.353621    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13313afc-657b-4a56-a50d-e0547d6781de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-607bc420-bhtbg\" (UID: \"13313afc-657b-4a56-a50d-e0547d6781de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607bc420-bhtbg"
Oct 31 03:40:39.417623 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:39.417591    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrj67\" (UniqueName: \"kubernetes.io/projected/13313afc-657b-4a56-a50d-e0547d6781de-kube-api-access-zrj67\") pod \"maas-api-revoke-sub-607bc420-bhtbg\" (UID: \"13313afc-657b-4a56-a50d-e0547d6781de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607bc420-bhtbg"
Oct 31 03:40:39.520479 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:39.520442    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607bc420-bhtbg"
Oct 31 03:40:39.918702 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:39.918668    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-607bc420-bhtbg"]
Oct 31 03:40:39.919336 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:40:39.919302    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13313afc_657b_4a56_a50d_e0547d6781de.slice/crio-8700277f7042e0ffac99f9b085db9080b574de7766ced2594f55f4a8fec56ed4 WatchSource:0}: Error finding container 8700277f7042e0ffac99f9b085db9080b574de7766ced2594f55f4a8fec56ed4: Status 404 returned error can't find the container with id 8700277f7042e0ffac99f9b085db9080b574de7766ced2594f55f4a8fec56ed4
Oct 31 03:40:40.414224 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:40.414188    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607bc420-bhtbg" event={"ID":"13313afc-657b-4a56-a50d-e0547d6781de","Type":"ContainerStarted","Data":"e7dbbf927833729e6d2139c2e95d80020c4900b22c0851be91fe8b3cf1abd150"}
Oct 31 03:40:40.414690 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:40.414282    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607bc420-bhtbg" event={"ID":"13313afc-657b-4a56-a50d-e0547d6781de","Type":"ContainerStarted","Data":"8700277f7042e0ffac99f9b085db9080b574de7766ced2594f55f4a8fec56ed4"}
Oct 31 03:40:40.530815 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:40.530755    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607bc420-bhtbg" podStartSLOduration=1.530742743 podStartE2EDuration="1.530742743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:40.53001432 +0000 UTC m=+1266.040996710" watchObservedRunningTime="2026-10-31 03:40:40.530742743 +0000 UTC m=+1266.041725173"
Oct 31 03:40:41.423677 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:41.423637    4447 generic.go:356] "Generic (PLEG): container finished" podID="13313afc-657b-4a56-a50d-e0547d6781de" containerID="e7dbbf927833729e6d2139c2e95d80020c4900b22c0851be91fe8b3cf1abd150" exitCode=0
Oct 31 03:40:41.424138 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:41.423676    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607bc420-bhtbg" event={"ID":"13313afc-657b-4a56-a50d-e0547d6781de","Type":"ContainerDied","Data":"e7dbbf927833729e6d2139c2e95d80020c4900b22c0851be91fe8b3cf1abd150"}
Oct 31 03:40:42.610369 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:42.610342    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607bc420-bhtbg"
Oct 31 03:40:42.702182 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:42.702105    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13313afc-657b-4a56-a50d-e0547d6781de-kube-api-access-zrj67\" (UniqueName: \"kubernetes.io/projected/13313afc-657b-4a56-a50d-e0547d6781de-kube-api-access-zrj67\") pod \"13313afc-657b-4a56-a50d-e0547d6781de\" (UID: \"13313afc-657b-4a56-a50d-e0547d6781de\") "
Oct 31 03:40:42.702336 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:42.702283    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13313afc-657b-4a56-a50d-e0547d6781de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13313afc-657b-4a56-a50d-e0547d6781de-maas-api-openshift-ca\") pod \"13313afc-657b-4a56-a50d-e0547d6781de\" (UID: \"13313afc-657b-4a56-a50d-e0547d6781de\") "
Oct 31 03:40:42.702518 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:42.702499    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13313afc-657b-4a56-a50d-e0547d6781de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13313afc-657b-4a56-a50d-e0547d6781de-maas-api-xks-ca\") pod \"13313afc-657b-4a56-a50d-e0547d6781de\" (UID: \"13313afc-657b-4a56-a50d-e0547d6781de\") "
Oct 31 03:40:42.702626 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:42.702604    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13313afc-657b-4a56-a50d-e0547d6781de-maas-api-openshift-ca" pod "13313afc-657b-4a56-a50d-e0547d6781de" (UID: "13313afc-657b-4a56-a50d-e0547d6781de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:40:42.703003 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:42.702985    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13313afc-657b-4a56-a50d-e0547d6781de-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:40:42.704606 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:42.704581    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13313afc-657b-4a56-a50d-e0547d6781de-maas-api-xks-ca" pod "13313afc-657b-4a56-a50d-e0547d6781de" (UID: "13313afc-657b-4a56-a50d-e0547d6781de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:40:42.704701 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:42.704658    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13313afc-657b-4a56-a50d-e0547d6781de-kube-api-access-zrj67" pod "13313afc-657b-4a56-a50d-e0547d6781de" (UID: "13313afc-657b-4a56-a50d-e0547d6781de"). InnerVolumeSpecName "kube-api-access-zrj67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:40:42.804737 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:42.804695    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13313afc-657b-4a56-a50d-e0547d6781de-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:40:42.804737 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:42.804730    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrj67\" (UniqueName: \"kubernetes.io/projected/13313afc-657b-4a56-a50d-e0547d6781de-kube-api-access-zrj67\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:40:43.440413 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:43.440382    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607bc420-bhtbg"
Oct 31 03:40:43.440413 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:43.440400    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607bc420-bhtbg" event={"ID":"13313afc-657b-4a56-a50d-e0547d6781de","Type":"ContainerDied","Data":"8700277f7042e0ffac99f9b085db9080b574de7766ced2594f55f4a8fec56ed4"}
Oct 31 03:40:43.440629 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:43.440470    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8700277f7042e0ffac99f9b085db9080b574de7766ced2594f55f4a8fec56ed4"
Oct 31 03:40:49.003292 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:40:49.003259    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:40:55.219456 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.219419    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"]
Oct 31 03:40:55.220167 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.220153    4447 cpu_manager.go:414] "containerMap: removing container" podUID="13313afc-657b-4a56-a50d-e0547d6781de" containerName="revoke-keys"
Oct 31 03:40:55.220240 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.220168    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13313afc-657b-4a56-a50d-e0547d6781de" containerName="revoke-keys"
Oct 31 03:40:55.220286 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.220277    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="13313afc-657b-4a56-a50d-e0547d6781de" containerName="revoke-keys"
Oct 31 03:40:55.223836 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.223816    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.270121 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.270091    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"]
Oct 31 03:40:55.274928 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.274906    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.277799 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.277773    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.277986 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.277808    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.277986 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.277836    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.277986 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.277972    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p24jk\" (UniqueName: \"kubernetes.io/projected/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-kube-api-access-p24jk\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.278174 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.278009    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.278229 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.278200    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.278306 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.278284    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.278422 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.278347    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.278422 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.278415    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.320850 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.320815    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"]
Oct 31 03:40:55.375262 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.375230    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"]
Oct 31 03:40:55.380367 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.380336    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.380517 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.380444    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.380517 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.380479    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.380655 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.380576    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.380716 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.380665    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.380716 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.380706    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.380817 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.380737    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.380817 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.380774    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.380817 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.380796    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p24jk\" (UniqueName: \"kubernetes.io/projected/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-kube-api-access-p24jk\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.381208 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.381184    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.381420 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.381401    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.381498 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.381450    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.381542 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.381520    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.381577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.381554    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.381622 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.381605    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4bzb\" (UniqueName: \"kubernetes.io/projected/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-kube-api-access-r4bzb\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.381736 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.381723    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.381799 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.381781    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.381862 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.381824    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.382818 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.382394    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.382818 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.382680    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.382818 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.382773    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.383084 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.382975    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.383084 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.383019    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.383992 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.383960    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.387145 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.387125    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.442958 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.442925    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p24jk\" (UniqueName: \"kubernetes.io/projected/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-kube-api-access-p24jk\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.443515 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.443493    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.484242 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.484160    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.484242 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.484204    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.484242 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.484232    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4bzb\" (UniqueName: \"kubernetes.io/projected/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-kube-api-access-r4bzb\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.484454 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.484356    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.484454 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.484386    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.484598 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.484565    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.484598 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.484588    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-data\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.484739 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.484624    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-workload-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.484739 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.484639    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.484739 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.484697    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-credential-socket\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.484739 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.484721    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.484945 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.484796    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.485040 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.485019    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-workload-certs\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.485478 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.485455    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istiod-ca-cert\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.486918 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.486898    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-envoy\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.486989 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.486953    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-podinfo\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.538720 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.538688    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:55.547122 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.547095    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-token\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.547250 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.547110    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4bzb\" (UniqueName: \"kubernetes.io/projected/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-kube-api-access-r4bzb\") pod \"e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") " pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.599317 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.599294    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:55.917255 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.917175    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"]
Oct 31 03:40:55.922999 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.922974    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:55.975347 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:40:55.975310    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c66081_9e3d_4097_8ba5_0624bcfcdc57.slice/crio-cfdbf47becf96e1da8f70910cbc7b3432a755c59f301d4b6b335edeba9191f93 WatchSource:0}: Error finding container cfdbf47becf96e1da8f70910cbc7b3432a755c59f301d4b6b335edeba9191f93: Status 404 returned error can't find the container with id cfdbf47becf96e1da8f70910cbc7b3432a755c59f301d4b6b335edeba9191f93
Oct 31 03:40:55.976082 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.976060    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"]
Oct 31 03:40:55.978521 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.978494    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:40:55.978605 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.978554    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:40:55.978605 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.978585    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:40:55.993334 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.993304    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:55.993520 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.993351    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:55.993597 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.993554    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:55.993656 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.993594    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:55.993755 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.993728    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:55.994011 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.993990    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80bc3946-0385-4655-a0eb-3266a188f7ab-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:55.994080 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.994045    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w7n6\" (UniqueName: \"kubernetes.io/projected/80bc3946-0385-4655-a0eb-3266a188f7ab-kube-api-access-9w7n6\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:55.994137 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.994085    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:55.994330 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:55.994306    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:56.011369 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.011339    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"]
Oct 31 03:40:56.029863 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.029832    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"]
Oct 31 03:40:56.035552 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:40:56.035526    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377bb5ad_a2ce_4d5a_a93d_bad2fc2dce3f.slice/crio-acb310576d31ca501f6c434b923cdfbbe2a870f6e3e8192f32d3e2904e3c5838 WatchSource:0}: Error finding container acb310576d31ca501f6c434b923cdfbbe2a870f6e3e8192f32d3e2904e3c5838: Status 404 returned error can't find the container with id acb310576d31ca501f6c434b923cdfbbe2a870f6e3e8192f32d3e2904e3c5838
Oct 31 03:40:56.038540 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.038500    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:40:56.038639 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.038599    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:40:56.038679 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.038645    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:40:56.096148 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.096110    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80bc3946-0385-4655-a0eb-3266a188f7ab-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:56.096310 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.096180    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w7n6\" (UniqueName: \"kubernetes.io/projected/80bc3946-0385-4655-a0eb-3266a188f7ab-kube-api-access-9w7n6\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:56.096310 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.096229    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:56.096436 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.096372    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:56.096436 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.096399    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:56.096436 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.096431    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:56.096838 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.096552    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:56.096838 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.096574    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:56.096838 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.096609    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:56.098098 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.097386    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:56.098098 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.097644    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:56.098098 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.097678    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:56.098098 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.097900    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:56.098098 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.098048    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80bc3946-0385-4655-a0eb-3266a188f7ab-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:56.099471 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.099445    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:56.100594 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.100570    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:56.159341 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.159313    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w7n6\" (UniqueName: \"kubernetes.io/projected/80bc3946-0385-4655-a0eb-3266a188f7ab-kube-api-access-9w7n6\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:56.159516 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.159355    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:56.238345 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.238315    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:56.293431 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.293395    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"]
Oct 31 03:40:56.300148 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.300121    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.327361 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.327328    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"]
Oct 31 03:40:56.332793 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.332771    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.367951 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.367407    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"]
Oct 31 03:40:56.413557 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.409637    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.413557 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.409697    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.413557 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.409771    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.413557 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.409798    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.413557 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.409834    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.413557 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.409856    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.413557 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.409943    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.413557 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.409984    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.413557 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.410034    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.413557 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.410107    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.413557 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.410137    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.413557 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.410182    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6xc2\" (UniqueName: \"kubernetes.io/projected/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-kube-api-access-p6xc2\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.413557 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.410228    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.413557 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.410279    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.413557 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.410417    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.413557 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.410664    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfwrr\" (UniqueName: \"kubernetes.io/projected/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-kube-api-access-rfwrr\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.413557 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.410787    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.413557 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.410858    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.419647 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.419622    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"]
Oct 31 03:40:56.512110 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.512016    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.512110 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.512055    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.512110 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.512105    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.512389 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.512122    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.512389 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.512152    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.512389 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.512173    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.512389 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.512228    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.512389 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.512263    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.512389 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.512311    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.512389 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.512384    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.512740 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.512411    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.512740 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.512444    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.512740 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.512453    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6xc2\" (UniqueName: \"kubernetes.io/projected/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-kube-api-access-p6xc2\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.512740 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.512489    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-workload-certs\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.512740 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.512529    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.512740 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.512582    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.512740 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.512630    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.512740 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.512677    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.512740 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.512722    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfwrr\" (UniqueName: \"kubernetes.io/projected/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-kube-api-access-rfwrr\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.513221 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.512774    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.513221 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.512825    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.513221 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.512872    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.513221 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.513172    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.513425 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.513358    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-credential-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.513593 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.513556    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.513706 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.513682    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-workload-socket\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.513770 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.513745    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-data\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.514090 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.514066    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istiod-ca-cert\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.515657 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.515635    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.516188 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.516164    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-envoy\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.516666 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.516642    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.517105 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.517082    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-podinfo\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.542644 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.542595    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4" event={"ID":"e3c66081-9e3d-4097-8ba5-0624bcfcdc57","Type":"ContainerStarted","Data":"b080e3ad729e8cd21ff47475185d585c0f3d35734542445ba17d0c6ed884a695"}
Oct 31 03:40:56.542820 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.542696    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4" event={"ID":"e3c66081-9e3d-4097-8ba5-0624bcfcdc57","Type":"ContainerStarted","Data":"cfdbf47becf96e1da8f70910cbc7b3432a755c59f301d4b6b335edeba9191f93"}
Oct 31 03:40:56.544946 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.544916    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs" event={"ID":"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f","Type":"ContainerStarted","Data":"bb0be6b368ff75326103c6413b3f785eeafcc65f4730fe6e5793d46baa2961a2"}
Oct 31 03:40:56.545089 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.544975    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs" event={"ID":"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f","Type":"ContainerStarted","Data":"acb310576d31ca501f6c434b923cdfbbe2a870f6e3e8192f32d3e2904e3c5838"}
Oct 31 03:40:56.574449 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.574416    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6xc2\" (UniqueName: \"kubernetes.io/projected/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-kube-api-access-p6xc2\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.579556 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.579520    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfwrr\" (UniqueName: \"kubernetes.io/projected/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-kube-api-access-rfwrr\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.579941 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.579861    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.586955 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.582312    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-token\") pod \"e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") " pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.599910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.599851    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:40:56.633102 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.630489    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:56.653984 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.652227    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"]
Oct 31 03:40:56.659229 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.659197    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:56.660550 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:40:56.660524    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80bc3946_0385_4655_a0eb_3266a188f7ab.slice/crio-50d6730da8def4ba8e959b60df66e4bba571899b8a4c40798f300adbc7b4ab17 WatchSource:0}: Error finding container 50d6730da8def4ba8e959b60df66e4bba571899b8a4c40798f300adbc7b4ab17: Status 404 returned error can't find the container with id 50d6730da8def4ba8e959b60df66e4bba571899b8a4c40798f300adbc7b4ab17
Oct 31 03:40:56.664173 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.663569    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:40:56.664173 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.663636    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:40:56.664173 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.663663    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:40:56.736087 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.732243    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs" podStartSLOduration=1.7322234619999999 podStartE2EDuration="1.732223462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:56.712403987 +0000 UTC m=+1282.223386365" watchObservedRunningTime="2026-10-31 03:40:56.732223462 +0000 UTC m=+1282.243205840"
Oct 31 03:40:56.859984 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:56.851189    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4" podStartSLOduration=1.851169009 podStartE2EDuration="1.851169009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:56.846638827 +0000 UTC m=+1282.357621226" watchObservedRunningTime="2026-10-31 03:40:56.851169009 +0000 UTC m=+1282.362151387"
Oct 31 03:40:57.155179 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:40:57.152423    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e5b912_f46c_4ce3_ad7e_521e506e5e88.slice/crio-e1548d9a584aa6decb193adc88afc3a2002b7f2f057dbfb9b46dd111667ab8ef WatchSource:0}: Error finding container e1548d9a584aa6decb193adc88afc3a2002b7f2f057dbfb9b46dd111667ab8ef: Status 404 returned error can't find the container with id e1548d9a584aa6decb193adc88afc3a2002b7f2f057dbfb9b46dd111667ab8ef
Oct 31 03:40:57.155179 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:57.155112    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"]
Oct 31 03:40:57.158454 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:57.157177    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:40:57.158454 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:57.158128    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:40:57.158454 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:57.158183    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:40:57.423968 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:57.422916    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"]
Oct 31 03:40:57.448515 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:57.448226    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:40:57.449601 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:57.448738    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:40:57.449601 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:57.448793    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:40:57.543203 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:57.543121    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:40:57.545648 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:57.545620    4447 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 03:40:57.545842 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:57.545816    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4" podUID="e3c66081-9e3d-4097-8ba5-0624bcfcdc57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 03:40:57.589946 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:57.589224    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz" event={"ID":"d379c75c-04e8-4ab1-8b40-e8de8b0631ef","Type":"ContainerStarted","Data":"76c5dd4e58678ac2091d1848f72e290c9e74d3f165d91612bb51a2fea8207cb5"}
Oct 31 03:40:57.638912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:57.634273    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5" event={"ID":"f1e5b912-f46c-4ce3-ad7e-521e506e5e88","Type":"ContainerStarted","Data":"0d37ad73241bf95ccbc852fe988a0f6bc29dfd5e747bf6ac8faf70d99b6077d2"}
Oct 31 03:40:57.638912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:57.634398    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5" event={"ID":"f1e5b912-f46c-4ce3-ad7e-521e506e5e88","Type":"ContainerStarted","Data":"e1548d9a584aa6decb193adc88afc3a2002b7f2f057dbfb9b46dd111667ab8ef"}
Oct 31 03:40:57.655512 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:57.654031    4447 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 03:40:57.655512 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:57.654095    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs" podUID="377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 03:40:57.667686 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:57.667649    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz" event={"ID":"80bc3946-0385-4655-a0eb-3266a188f7ab","Type":"ContainerStarted","Data":"795351b27ca0ba3a922bcf786f45a2cfa79947e03f4a28921288e4a4a90b36d7"}
Oct 31 03:40:57.667990 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:57.667964    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz" event={"ID":"80bc3946-0385-4655-a0eb-3266a188f7ab","Type":"ContainerStarted","Data":"50d6730da8def4ba8e959b60df66e4bba571899b8a4c40798f300adbc7b4ab17"}
Oct 31 03:40:57.952973 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:57.923003    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5" podStartSLOduration=1.922982574 podStartE2EDuration="1.922982574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:57.782431922 +0000 UTC m=+1283.293414300" watchObservedRunningTime="2026-10-31 03:40:57.922982574 +0000 UTC m=+1283.433964951"
Oct 31 03:40:58.250918 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:58.243177    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:40:58.277785 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:58.274027    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 03:40:58.277785 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:58.274091    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz" podUID="80bc3946-0385-4655-a0eb-3266a188f7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 03:40:58.541968 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:58.541020    4447 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 03:40:58.541968 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:58.541078    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4" podUID="e3c66081-9e3d-4097-8ba5-0624bcfcdc57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 03:40:58.628911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:58.628283    4447 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 03:40:58.628911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:58.628355    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs" podUID="377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 03:40:58.651188 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:58.651149    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:40:58.651451 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:58.651424    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5" podUID="f1e5b912-f46c-4ce3-ad7e-521e506e5e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 03:40:58.949704 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:58.949579    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz" podStartSLOduration=3.949554248 podStartE2EDuration="3.949554248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:57.953923975 +0000 UTC m=+1283.464906352" watchObservedRunningTime="2026-10-31 03:40:58.949554248 +0000 UTC m=+1284.460536628"
Oct 31 03:40:58.950515 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:58.950483    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:40:58.950640 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:58.950560    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz" event={"ID":"d379c75c-04e8-4ab1-8b40-e8de8b0631ef","Type":"ContainerStarted","Data":"b698f8298b01106fb48ff2ff22b84a4a816f68fb1099fee487f1c1bb14730390"}
Oct 31 03:40:58.952679 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:58.952628    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz" podStartSLOduration=2.952610234 podStartE2EDuration="2.952610234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:58.919589219 +0000 UTC m=+1284.430571609" watchObservedRunningTime="2026-10-31 03:40:58.952610234 +0000 UTC m=+1284.463592613"
Oct 31 03:40:59.242474 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:59.242439    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 03:40:59.242794 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:59.242748    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz" podUID="80bc3946-0385-4655-a0eb-3266a188f7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 03:40:59.541501 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:59.541401    4447 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 03:40:59.542308 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:59.541739    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4" podUID="e3c66081-9e3d-4097-8ba5-0624bcfcdc57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 03:40:59.603777 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:59.603744    4447 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 03:40:59.604528 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:59.604499    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs" podUID="377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 03:40:59.640209 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:59.640084    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:40:59.640209 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:59.640139    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5" podUID="f1e5b912-f46c-4ce3-ad7e-521e506e5e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 03:40:59.682399 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:59.679637    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:40:59.682736 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:59.682705    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:40:59.682959 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:59.682929    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz" podUID="d379c75c-04e8-4ab1-8b40-e8de8b0631ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 03:40:59.835978 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:59.826188    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" podUID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:40:59.899945 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:40:59.899378    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:00.066917 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:00.065446    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:00.267105 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:00.267052    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 03:41:00.269824 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:00.269685    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz" podUID="80bc3946-0385-4655-a0eb-3266a188f7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 03:41:00.323897 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:00.323029    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:00.599967 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:00.579343    4447 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 03:41:00.599967 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:00.579430    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4" podUID="e3c66081-9e3d-4097-8ba5-0624bcfcdc57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 03:41:00.808912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:00.807177    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:41:00.808912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:00.807256    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5" podUID="f1e5b912-f46c-4ce3-ad7e-521e506e5e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 03:41:00.860192 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:00.835371    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:41:00.860192 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:00.835442    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz" podUID="d379c75c-04e8-4ab1-8b40-e8de8b0631ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 03:41:00.860192 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:00.841222    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded"
Oct 31 03:41:00.860192 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:00.858914    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" podUID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:00.860454 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:00.860209    4447 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 03:41:00.860454 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:00.860255    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs" podUID="377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 03:41:00.910911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:00.904971    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:01.115965 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:01.115327    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:01.244041 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:01.244005    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 03:41:01.251090 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:01.251049    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz" podUID="80bc3946-0385-4655-a0eb-3266a188f7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 03:41:01.346911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:01.307064    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:41:01.357050 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:01.357010    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:41:01.357313 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:01.357300    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:41:01.399364 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:01.347200    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:01.441922 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:01.426021    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:41:01.573912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:01.570048    4447 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 03:41:01.589038 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:01.588988    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4" podUID="e3c66081-9e3d-4097-8ba5-0624bcfcdc57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 03:41:01.677210 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:01.644983    4447 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 03:41:01.677210 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:01.645044    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs" podUID="377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 03:41:01.677210 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:01.673113    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:41:01.677210 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:01.673171    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz" podUID="d379c75c-04e8-4ab1-8b40-e8de8b0631ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 03:41:01.677210 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:01.676405    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:41:01.677210 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:01.676454    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5" podUID="f1e5b912-f46c-4ce3-ad7e-521e506e5e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 03:41:01.853914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:01.849187    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:01.871098 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:01.861975    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" podUID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:01.871098 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:01.862128    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:41:01.871098 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:01.864403    4447 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:01.907433 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:01.907389    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:02.138262 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:02.138215    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:02.312913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:02.287047    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 03:41:02.459972 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:02.316062    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz" podUID="80bc3946-0385-4655-a0eb-3266a188f7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 03:41:02.459972 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:02.436141    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:02.903985 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:02.900432    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:03.081911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:02.993599    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:41:03.081911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.050418    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz" podUID="d379c75c-04e8-4ab1-8b40-e8de8b0631ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 03:41:03.081911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.000470    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:41:03.081911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.050474    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5" podUID="f1e5b912-f46c-4ce3-ad7e-521e506e5e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 03:41:03.081911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.062521    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:03.090262 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.089037    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" podUID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:41:03.148459 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.146937    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 31 03:41:03.220442 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.217112    4447 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 03:41:03.220442 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.217168    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4" podUID="e3c66081-9e3d-4097-8ba5-0624bcfcdc57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 03:41:03.248991 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.248954    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 03:41:03.249273 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.249243    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz" podUID="80bc3946-0385-4655-a0eb-3266a188f7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 03:41:03.321912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.302016    4447 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 03:41:03.321912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.302077    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs" podUID="377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 03:41:03.346969 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.346930    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7168-211ae9"]
Oct 31 03:41:03.441905 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.441844    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:03.579004 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.547340    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7168-211ae9"
Oct 31 03:41:03.620769 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.558667    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7168-211ae9"]
Oct 31 03:41:03.620769 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.600938    4447 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 03:41:03.620769 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.620255    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs" podUID="377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 03:41:03.653071 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.652473    4447 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 03:41:03.653071 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.652528    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4" podUID="e3c66081-9e3d-4097-8ba5-0624bcfcdc57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 03:41:03.653071 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.652979    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:41:03.653071 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.653016    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5" podUID="f1e5b912-f46c-4ce3-ad7e-521e506e5e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 03:41:03.685914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.660426    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:41:03.685914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.660483    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz" podUID="d379c75c-04e8-4ab1-8b40-e8de8b0631ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 03:41:03.710908 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.693563    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mww68\" (UniqueName: \"kubernetes.io/projected/af979b05-aed5-4a7a-acd5-953fb5d5c456-kube-api-access-mww68\") pod \"test-curl-7168-211ae9\" (UID: \"af979b05-aed5-4a7a-acd5-953fb5d5c456\") " pod="opendatahub/test-curl-7168-211ae9"
Oct 31 03:41:03.783485 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.783454    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hppm7\""
Oct 31 03:41:03.865967 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:03.865174    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mww68\" (UniqueName: \"kubernetes.io/projected/af979b05-aed5-4a7a-acd5-953fb5d5c456-kube-api-access-mww68\") pod \"test-curl-7168-211ae9\" (UID: \"af979b05-aed5-4a7a-acd5-953fb5d5c456\") " pod="opendatahub/test-curl-7168-211ae9"
Oct 31 03:41:04.054910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:04.053975    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:04.090199 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:04.090141    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" podUID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:04.090871 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:04.090343    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 31 03:41:04.179951 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:04.179796    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:04.262911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:04.259549    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mww68\" (UniqueName: \"kubernetes.io/projected/af979b05-aed5-4a7a-acd5-953fb5d5c456-kube-api-access-mww68\") pod \"test-curl-7168-211ae9\" (UID: \"af979b05-aed5-4a7a-acd5-953fb5d5c456\") " pod="opendatahub/test-curl-7168-211ae9"
Oct 31 03:41:04.267027 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:04.262296    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 03:41:04.267296 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:04.267262    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz" podUID="80bc3946-0385-4655-a0eb-3266a188f7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 03:41:04.353911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:04.343085    4447 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:04.424301 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:04.424258    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7168-211ae9"
Oct 31 03:41:04.545937 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:04.541873    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:04.669934 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:04.641713    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:41:04.669934 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:04.664325    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5" podUID="f1e5b912-f46c-4ce3-ad7e-521e506e5e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 03:41:04.756044 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:04.756004    4447 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 03:41:04.756287 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:04.756256    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs" podUID="377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 03:41:04.782202 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:04.760683    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:41:04.831905 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:04.831845    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz" podUID="d379c75c-04e8-4ab1-8b40-e8de8b0631ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 03:41:04.882696 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:04.882660    4447 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 03:41:04.882991 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:04.882929    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4" podUID="e3c66081-9e3d-4097-8ba5-0624bcfcdc57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 03:41:05.100970 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:05.082656    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:05.102925 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:05.102865    4447 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded"
Oct 31 03:41:05.147344 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:05.144915    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:05.168925 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:05.161043    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" podUID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:41:05.204178 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:05.204130    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:05.232528 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:05.232484    4447 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:05.282913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:05.280117    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 03:41:05.282913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:05.280187    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz" podUID="80bc3946-0385-4655-a0eb-3266a188f7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 03:41:05.610073 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:05.567035    4447 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 03:41:05.610073 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:05.567101    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4" podUID="e3c66081-9e3d-4097-8ba5-0624bcfcdc57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 03:41:05.610073 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:05.585463    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 03:41:05.636547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:05.614628    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:41:05.706908 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:05.670092    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:41:06.018922 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:06.018286    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5" podUID="f1e5b912-f46c-4ce3-ad7e-521e506e5e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 03:41:06.110577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:06.080353    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:41:06.110577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:06.080408    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz" podUID="d379c75c-04e8-4ab1-8b40-e8de8b0631ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 03:41:06.199499 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:06.170013    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:06.199499 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:06.170457    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" podUID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:06.199499 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:06.170730    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 31 03:41:06.474361 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:06.472754    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:41:06.509083 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:06.482021    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:41:06.545155 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:06.545118    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 03:41:06.545449 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:06.545409    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz" podUID="80bc3946-0385-4655-a0eb-3266a188f7ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 03:41:06.572327 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:06.572297    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:41:06.616316 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:06.616283    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:41:06.649233 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:06.648547    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:41:06.649233 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:06.648606    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5" podUID="f1e5b912-f46c-4ce3-ad7e-521e506e5e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 03:41:06.726649 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:06.726563    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:41:06.728019 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:06.727962    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz" podUID="d379c75c-04e8-4ab1-8b40-e8de8b0631ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 03:41:06.748862 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:06.748804    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:06.812496 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:06.808096    4447 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:41:06.812496 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:06.811157    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7168-211ae9"]
Oct 31 03:41:06.821346 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:06.821304    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:41:06.898907 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:06.882476    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:41:06.911249 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:06.911219    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:41:07.078288 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:07.078188    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:41:07.096524 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:07.084813    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:41:07.101921 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:07.098716    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:41:07.545237 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:07.544797    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:41:07.548455 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:07.548329    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:41:07.549899 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:07.549851    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:41:07.635307 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:07.633485    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:41:07.635307 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:07.634597    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5" podUID="f1e5b912-f46c-4ce3-ad7e-521e506e5e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 03:41:07.666867 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:07.661692    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:41:07.666867 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:07.661764    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz" podUID="d379c75c-04e8-4ab1-8b40-e8de8b0631ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 31 03:41:07.764776 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:07.764740    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7168-211ae9" event={"ID":"af979b05-aed5-4a7a-acd5-953fb5d5c456","Type":"ContainerStarted","Data":"53c599ac85fa1219a9fd9de4c22c7758ea36ff4c9d536c9eec59612899dd3ba0"}
Oct 31 03:41:08.703909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:08.687174    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5165bf8-jb2d9"]
Oct 31 03:41:08.721945 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:08.709807    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5165bf8-jb2d9"
Oct 31 03:41:08.752833 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:08.752461    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5165bf8-jb2d9"]
Oct 31 03:41:08.766673 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:08.766630    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 03:41:08.767326 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:08.767306    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:41:08.783928 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:08.783873    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:41:08.785816 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:08.785642    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:41:08.787186 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:08.787122    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7168-211ae9" event={"ID":"af979b05-aed5-4a7a-acd5-953fb5d5c456","Type":"ContainerStarted","Data":"2db4138322966323f4ffcb3d1cacf65a3f8284d16e6e89f8b97ad619b4f95a3c"}
Oct 31 03:41:08.789484 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:08.789407    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:41:08.826522 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:08.825995    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn9j9\" (UniqueName: \"kubernetes.io/projected/1ade9412-157f-4df2-8928-117d91e4480c-kube-api-access-jn9j9\") pod \"maas-api-revoke-sub-a5165bf8-jb2d9\" (UID: \"1ade9412-157f-4df2-8928-117d91e4480c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5165bf8-jb2d9"
Oct 31 03:41:08.826522 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:08.826044    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ade9412-157f-4df2-8928-117d91e4480c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5165bf8-jb2d9\" (UID: \"1ade9412-157f-4df2-8928-117d91e4480c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5165bf8-jb2d9"
Oct 31 03:41:08.826522 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:08.826257    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ade9412-157f-4df2-8928-117d91e4480c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5165bf8-jb2d9\" (UID: \"1ade9412-157f-4df2-8928-117d91e4480c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5165bf8-jb2d9"
Oct 31 03:41:08.901075 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:08.900963    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:41:08.901677 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:08.901661    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:41:08.903594 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:08.903554    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:41:08.927993 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:08.927954    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn9j9\" (UniqueName: \"kubernetes.io/projected/1ade9412-157f-4df2-8928-117d91e4480c-kube-api-access-jn9j9\") pod \"maas-api-revoke-sub-a5165bf8-jb2d9\" (UID: \"1ade9412-157f-4df2-8928-117d91e4480c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5165bf8-jb2d9"
Oct 31 03:41:08.927993 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:08.927991    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ade9412-157f-4df2-8928-117d91e4480c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5165bf8-jb2d9\" (UID: \"1ade9412-157f-4df2-8928-117d91e4480c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5165bf8-jb2d9"
Oct 31 03:41:08.928193 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:08.928104    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ade9412-157f-4df2-8928-117d91e4480c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5165bf8-jb2d9\" (UID: \"1ade9412-157f-4df2-8928-117d91e4480c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5165bf8-jb2d9"
Oct 31 03:41:08.928845 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:08.928822    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ade9412-157f-4df2-8928-117d91e4480c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5165bf8-jb2d9\" (UID: \"1ade9412-157f-4df2-8928-117d91e4480c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5165bf8-jb2d9"
Oct 31 03:41:08.930601 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:08.930581    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ade9412-157f-4df2-8928-117d91e4480c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5165bf8-jb2d9\" (UID: \"1ade9412-157f-4df2-8928-117d91e4480c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5165bf8-jb2d9"
Oct 31 03:41:09.015039 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:09.014955    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn9j9\" (UniqueName: \"kubernetes.io/projected/1ade9412-157f-4df2-8928-117d91e4480c-kube-api-access-jn9j9\") pod \"maas-api-revoke-sub-a5165bf8-jb2d9\" (UID: \"1ade9412-157f-4df2-8928-117d91e4480c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5165bf8-jb2d9"
Oct 31 03:41:09.070927 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:09.070893    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5165bf8-jb2d9"
Oct 31 03:41:09.131563 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:09.131502    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7168-211ae9" podStartSLOduration=5.8846793349999995 podStartE2EDuration="7.131479825s" totalImagesPullingTime="1.24680049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:09.106460109 +0000 UTC m=+1294.617442487" watchObservedRunningTime="2026-10-31 03:41:09.131479825 +0000 UTC m=+1294.642462203"
Oct 31 03:41:09.525808 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:41:09.525766    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ade9412_157f_4df2_8928_117d91e4480c.slice/crio-f0b6d3c1871f3dc1c0fadf1af0de1df8386dacddb9e272197b6e457ba4d63998 WatchSource:0}: Error finding container f0b6d3c1871f3dc1c0fadf1af0de1df8386dacddb9e272197b6e457ba4d63998: Status 404 returned error can't find the container with id f0b6d3c1871f3dc1c0fadf1af0de1df8386dacddb9e272197b6e457ba4d63998
Oct 31 03:41:09.527763 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:09.527738    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5165bf8-jb2d9"]
Oct 31 03:41:09.816959 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:09.814831    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5165bf8-jb2d9" event={"ID":"1ade9412-157f-4df2-8928-117d91e4480c","Type":"ContainerStarted","Data":"f0b6d3c1871f3dc1c0fadf1af0de1df8386dacddb9e272197b6e457ba4d63998"}
Oct 31 03:41:11.018212 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:10.985475    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5165bf8-jb2d9" event={"ID":"1ade9412-157f-4df2-8928-117d91e4480c","Type":"ContainerStarted","Data":"15f977929ea6e7ac9a2e4e040a83b945d255d6b51d2a94ce7ef068997888e1aa"}
Oct 31 03:41:11.101914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:11.098800    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5165bf8-jb2d9" podStartSLOduration=3.098772601 podStartE2EDuration="3.098772601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:11.078380184 +0000 UTC m=+1296.589362559" watchObservedRunningTime="2026-10-31 03:41:11.098772601 +0000 UTC m=+1296.609755031"
Oct 31 03:41:11.854910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:11.853962    4447 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:11.927911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:11.924420    4447 generic.go:356] "Generic (PLEG): container finished" podID="1ade9412-157f-4df2-8928-117d91e4480c" containerID="15f977929ea6e7ac9a2e4e040a83b945d255d6b51d2a94ce7ef068997888e1aa" exitCode=0
Oct 31 03:41:11.927911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:11.924547    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5165bf8-jb2d9" event={"ID":"1ade9412-157f-4df2-8928-117d91e4480c","Type":"ContainerDied","Data":"15f977929ea6e7ac9a2e4e040a83b945d255d6b51d2a94ce7ef068997888e1aa"}
Oct 31 03:41:12.038922 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:12.037990    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:12.063910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:12.055470    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:12.095035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:12.094987    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" podUID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:12.151137 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:12.151044    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 31 03:41:12.457974 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:12.447223    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:13.089032 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:13.088983    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded"
Oct 31 03:41:13.123742 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:13.091588    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:13.147500 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:13.147466    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:41:13.147753 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:13.147737    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:41:13.151914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:13.150913    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" podUID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:41:13.151914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:13.151246    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:41:13.169360 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:13.151867    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:13.169360 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:13.168380    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:41:13.500909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:13.494066    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 03:41:13.528207 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:13.528178    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:41:14.162015 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:14.124003    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:14.162015 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:14.153053    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 31 03:41:14.163269 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:14.163227    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" podUID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:14.177010 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:14.170989    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 31 03:41:14.425967 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:14.390985    4447 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:14.542036 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:14.541051    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 03:41:15.169090 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:15.169032    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" podUID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:15.316979 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:15.316922    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded"
Oct 31 03:41:15.330419 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:15.330361    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 31 03:41:15.513703 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:15.342398    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 31 03:41:15.514840 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:15.342695    4447 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:15.881082 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:15.880977    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-s6nfq" podUID="41c075e6-b57b-4e72-ad58-59450e64c8a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:15.926706 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:15.922274    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xl2pv" podUID="a3767bbe-f08f-47b5-84eb-f01ebf697a41" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 03:41:15.926706 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:15.922621    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb" podUID="2b7caa8c-a0aa-4fbd-8918-288e39fc4f56" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Oct 31 03:41:15.998081 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:15.991209    4447 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded"
Oct 31 03:41:16.076059 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.075942    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:16.176021 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.175118    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" podUID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:16.289471 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:41:16.287764    4447 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.433s"
Oct 31 03:41:16.289471 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.287849    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:41:16.289471 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.288102    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:41:16.289471 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.288267    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:41:16.289471 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.288442    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:41:16.290411 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.290368    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:41:16.306342 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.299937    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5165bf8-jb2d9"
Oct 31 03:41:16.416966 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.416930    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7168-211ae9"]
Oct 31 03:41:16.417178 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.417144    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7168-211ae9" podUID="af979b05-aed5-4a7a-acd5-953fb5d5c456" containerName="test-curl-7168-211ae9" containerID="cri-o://2db4138322966323f4ffcb3d1cacf65a3f8284d16e6e89f8b97ad619b4f95a3c" gracePeriod=2
Oct 31 03:41:16.421490 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.421463    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7168-211ae9"]
Oct 31 03:41:16.453914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.453862    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ade9412-157f-4df2-8928-117d91e4480c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ade9412-157f-4df2-8928-117d91e4480c-maas-api-xks-ca\") pod \"1ade9412-157f-4df2-8928-117d91e4480c\" (UID: \"1ade9412-157f-4df2-8928-117d91e4480c\") "
Oct 31 03:41:16.454061 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.453959    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ade9412-157f-4df2-8928-117d91e4480c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ade9412-157f-4df2-8928-117d91e4480c-maas-api-openshift-ca\") pod \"1ade9412-157f-4df2-8928-117d91e4480c\" (UID: \"1ade9412-157f-4df2-8928-117d91e4480c\") "
Oct 31 03:41:16.454061 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.454022    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ade9412-157f-4df2-8928-117d91e4480c-kube-api-access-jn9j9\" (UniqueName: \"kubernetes.io/projected/1ade9412-157f-4df2-8928-117d91e4480c-kube-api-access-jn9j9\") pod \"1ade9412-157f-4df2-8928-117d91e4480c\" (UID: \"1ade9412-157f-4df2-8928-117d91e4480c\") "
Oct 31 03:41:16.456686 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.456656    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ade9412-157f-4df2-8928-117d91e4480c-maas-api-openshift-ca" pod "1ade9412-157f-4df2-8928-117d91e4480c" (UID: "1ade9412-157f-4df2-8928-117d91e4480c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:41:16.459192 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.459156    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ade9412-157f-4df2-8928-117d91e4480c-maas-api-xks-ca" pod "1ade9412-157f-4df2-8928-117d91e4480c" (UID: "1ade9412-157f-4df2-8928-117d91e4480c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:41:16.459380 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.459354    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ade9412-157f-4df2-8928-117d91e4480c-kube-api-access-jn9j9" pod "1ade9412-157f-4df2-8928-117d91e4480c" (UID: "1ade9412-157f-4df2-8928-117d91e4480c"). InnerVolumeSpecName "kube-api-access-jn9j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:41:16.555898 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.555852    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ade9412-157f-4df2-8928-117d91e4480c-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:41:16.555898 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.555897    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn9j9\" (UniqueName: \"kubernetes.io/projected/1ade9412-157f-4df2-8928-117d91e4480c-kube-api-access-jn9j9\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:41:16.556073 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.555909    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ade9412-157f-4df2-8928-117d91e4480c-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:41:16.580125 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.580092    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7168-14a48a"]
Oct 31 03:41:16.582053 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.581300    4447 cpu_manager.go:414] "containerMap: removing container" podUID="1ade9412-157f-4df2-8928-117d91e4480c" containerName="revoke-keys"
Oct 31 03:41:16.582053 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.581323    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ade9412-157f-4df2-8928-117d91e4480c" containerName="revoke-keys"
Oct 31 03:41:16.582053 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.581393    4447 cpu_manager.go:414] "containerMap: removing container" podUID="af979b05-aed5-4a7a-acd5-953fb5d5c456" containerName="test-curl-7168-211ae9"
Oct 31 03:41:16.582053 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.581402    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af979b05-aed5-4a7a-acd5-953fb5d5c456" containerName="test-curl-7168-211ae9"
Oct 31 03:41:16.582053 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.581559    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="af979b05-aed5-4a7a-acd5-953fb5d5c456" containerName="test-curl-7168-211ae9"
Oct 31 03:41:16.582053 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.581577    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ade9412-157f-4df2-8928-117d91e4480c" containerName="revoke-keys"
Oct 31 03:41:16.586699 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.586674    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7168-14a48a"
Oct 31 03:41:16.668775 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.666832    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwfvt\" (UniqueName: \"kubernetes.io/projected/59ebedaa-2d25-4f2f-98df-22115b365656-kube-api-access-jwfvt\") pod \"test-curl-7168-14a48a\" (UID: \"59ebedaa-2d25-4f2f-98df-22115b365656\") " pod="opendatahub/test-curl-7168-14a48a"
Oct 31 03:41:16.702858 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.702827    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7168-14a48a"]
Oct 31 03:41:16.768153 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.768071    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwfvt\" (UniqueName: \"kubernetes.io/projected/59ebedaa-2d25-4f2f-98df-22115b365656-kube-api-access-jwfvt\") pod \"test-curl-7168-14a48a\" (UID: \"59ebedaa-2d25-4f2f-98df-22115b365656\") " pod="opendatahub/test-curl-7168-14a48a"
Oct 31 03:41:16.834067 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.834035    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwfvt\" (UniqueName: \"kubernetes.io/projected/59ebedaa-2d25-4f2f-98df-22115b365656-kube-api-access-jwfvt\") pod \"test-curl-7168-14a48a\" (UID: \"59ebedaa-2d25-4f2f-98df-22115b365656\") " pod="opendatahub/test-curl-7168-14a48a"
Oct 31 03:41:16.873812 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:41:16.873765    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7168-211ae9\" is forbidden: User \"system:node:ip-10-0-134-255.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-255.us-west-1.compute.internal' and this object" podUID="af979b05-aed5-4a7a-acd5-953fb5d5c456" pod="opendatahub/test-curl-7168-211ae9"
Oct 31 03:41:16.910104 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:16.905749    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7168-14a48a"
Oct 31 03:41:17.297015 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:17.287164    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5165bf8-jb2d9" event={"ID":"1ade9412-157f-4df2-8928-117d91e4480c","Type":"ContainerDied","Data":"f0b6d3c1871f3dc1c0fadf1af0de1df8386dacddb9e272197b6e457ba4d63998"}
Oct 31 03:41:17.297015 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:17.287233    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0b6d3c1871f3dc1c0fadf1af0de1df8386dacddb9e272197b6e457ba4d63998"
Oct 31 03:41:17.297015 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:17.287305    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5165bf8-jb2d9"
Oct 31 03:41:17.342566 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:17.342537    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7168-14a48a"]
Oct 31 03:41:18.319378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:18.319337    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7168-14a48a" event={"ID":"59ebedaa-2d25-4f2f-98df-22115b365656","Type":"ContainerStarted","Data":"055998e078e377788acca967b68e12e4c98d34f0daed9c07fb21b1d3ed53c683"}
Oct 31 03:41:18.319378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:18.319390    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7168-14a48a" event={"ID":"59ebedaa-2d25-4f2f-98df-22115b365656","Type":"ContainerStarted","Data":"c2e2993b078316ba07bcabb1329065845dfde4a29de865690a97a80dc39d16db"}
Oct 31 03:41:18.442297 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:18.438125    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7168-14a48a" podStartSLOduration=2.438101309 podStartE2EDuration="2.438101309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:18.433286693 +0000 UTC m=+1303.944269069" watchObservedRunningTime="2026-10-31 03:41:18.438101309 +0000 UTC m=+1303.949083686"
Oct 31 03:41:18.716499 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:18.716471    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7168-211ae9"
Oct 31 03:41:18.794916 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:18.789640    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af979b05-aed5-4a7a-acd5-953fb5d5c456-kube-api-access-mww68\" (UniqueName: \"kubernetes.io/projected/af979b05-aed5-4a7a-acd5-953fb5d5c456-kube-api-access-mww68\") pod \"af979b05-aed5-4a7a-acd5-953fb5d5c456\" (UID: \"af979b05-aed5-4a7a-acd5-953fb5d5c456\") "
Oct 31 03:41:18.819993 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:18.819956    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af979b05-aed5-4a7a-acd5-953fb5d5c456-kube-api-access-mww68" pod "af979b05-aed5-4a7a-acd5-953fb5d5c456" (UID: "af979b05-aed5-4a7a-acd5-953fb5d5c456"). InnerVolumeSpecName "kube-api-access-mww68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:41:18.917435 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:18.905241    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mww68\" (UniqueName: \"kubernetes.io/projected/af979b05-aed5-4a7a-acd5-953fb5d5c456-kube-api-access-mww68\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:41:19.005661 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:41:19.005610    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:41:19.333342 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:19.333306    4447 generic.go:356] "Generic (PLEG): container finished" podID="af979b05-aed5-4a7a-acd5-953fb5d5c456" containerID="2db4138322966323f4ffcb3d1cacf65a3f8284d16e6e89f8b97ad619b4f95a3c" exitCode=137
Oct 31 03:41:19.333752 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:19.333371    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7168-211ae9"
Oct 31 03:41:19.333752 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:19.333410    4447 scope.go:122] "RemoveContainer" containerID="2db4138322966323f4ffcb3d1cacf65a3f8284d16e6e89f8b97ad619b4f95a3c"
Oct 31 03:41:19.345154 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:19.345138    4447 scope.go:122] "RemoveContainer" containerID="2db4138322966323f4ffcb3d1cacf65a3f8284d16e6e89f8b97ad619b4f95a3c"
Oct 31 03:41:19.345470 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:41:19.345447    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2db4138322966323f4ffcb3d1cacf65a3f8284d16e6e89f8b97ad619b4f95a3c\": container with ID starting with 2db4138322966323f4ffcb3d1cacf65a3f8284d16e6e89f8b97ad619b4f95a3c not found: ID does not exist" containerID="2db4138322966323f4ffcb3d1cacf65a3f8284d16e6e89f8b97ad619b4f95a3c"
Oct 31 03:41:19.345534 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:19.345481    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2db4138322966323f4ffcb3d1cacf65a3f8284d16e6e89f8b97ad619b4f95a3c"} err="failed to get container status \"2db4138322966323f4ffcb3d1cacf65a3f8284d16e6e89f8b97ad619b4f95a3c\": rpc error: code = NotFound desc = could not find container \"2db4138322966323f4ffcb3d1cacf65a3f8284d16e6e89f8b97ad619b4f95a3c\": container with ID starting with 2db4138322966323f4ffcb3d1cacf65a3f8284d16e6e89f8b97ad619b4f95a3c not found: ID does not exist"
Oct 31 03:41:19.395475 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:41:19.395424    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7168-211ae9\" is forbidden: User \"system:node:ip-10-0-134-255.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-255.us-west-1.compute.internal' and this object" podUID="af979b05-aed5-4a7a-acd5-953fb5d5c456" pod="opendatahub/test-curl-7168-211ae9"
Oct 31 03:41:20.111840 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:20.111787    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7168-14a48a"]
Oct 31 03:41:20.112060 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:20.111878    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7168-14a48a"]
Oct 31 03:41:20.342706 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:20.342666    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7168-14a48a" podUID="59ebedaa-2d25-4f2f-98df-22115b365656" containerName="test-curl-7168-14a48a" containerID="cri-o://055998e078e377788acca967b68e12e4c98d34f0daed9c07fb21b1d3ed53c683" gracePeriod=2
Oct 31 03:41:20.394137 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:41:20.394047    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7168-211ae9\" is forbidden: User \"system:node:ip-10-0-134-255.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-255.us-west-1.compute.internal' and this object" podUID="af979b05-aed5-4a7a-acd5-953fb5d5c456" pod="opendatahub/test-curl-7168-211ae9"
Oct 31 03:41:20.445236 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:41:20.445199    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7168-14a48a\" is forbidden: User \"system:node:ip-10-0-134-255.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-255.us-west-1.compute.internal' and this object" podUID="59ebedaa-2d25-4f2f-98df-22115b365656" pod="opendatahub/test-curl-7168-14a48a"
Oct 31 03:41:20.610503 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:20.610469    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af979b05-aed5-4a7a-acd5-953fb5d5c456" path="/var/lib/kubelet/pods/af979b05-aed5-4a7a-acd5-953fb5d5c456/volumes"
Oct 31 03:41:22.635546 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:22.635520    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7168-14a48a"
Oct 31 03:41:22.738165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:22.738135    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ebedaa-2d25-4f2f-98df-22115b365656-kube-api-access-jwfvt\" (UniqueName: \"kubernetes.io/projected/59ebedaa-2d25-4f2f-98df-22115b365656-kube-api-access-jwfvt\") pod \"59ebedaa-2d25-4f2f-98df-22115b365656\" (UID: \"59ebedaa-2d25-4f2f-98df-22115b365656\") "
Oct 31 03:41:22.740690 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:22.740657    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ebedaa-2d25-4f2f-98df-22115b365656-kube-api-access-jwfvt" pod "59ebedaa-2d25-4f2f-98df-22115b365656" (UID: "59ebedaa-2d25-4f2f-98df-22115b365656"). InnerVolumeSpecName "kube-api-access-jwfvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:41:22.839827 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:22.839773    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwfvt\" (UniqueName: \"kubernetes.io/projected/59ebedaa-2d25-4f2f-98df-22115b365656-kube-api-access-jwfvt\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:41:23.383437 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:23.383405    4447 generic.go:356] "Generic (PLEG): container finished" podID="59ebedaa-2d25-4f2f-98df-22115b365656" containerID="055998e078e377788acca967b68e12e4c98d34f0daed9c07fb21b1d3ed53c683" exitCode=137
Oct 31 03:41:23.383599 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:23.383445    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7168-14a48a"
Oct 31 03:41:23.383599 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:23.383500    4447 scope.go:122] "RemoveContainer" containerID="055998e078e377788acca967b68e12e4c98d34f0daed9c07fb21b1d3ed53c683"
Oct 31 03:41:23.394466 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:23.394452    4447 scope.go:122] "RemoveContainer" containerID="055998e078e377788acca967b68e12e4c98d34f0daed9c07fb21b1d3ed53c683"
Oct 31 03:41:23.394733 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:41:23.394713    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"055998e078e377788acca967b68e12e4c98d34f0daed9c07fb21b1d3ed53c683\": container with ID starting with 055998e078e377788acca967b68e12e4c98d34f0daed9c07fb21b1d3ed53c683 not found: ID does not exist" containerID="055998e078e377788acca967b68e12e4c98d34f0daed9c07fb21b1d3ed53c683"
Oct 31 03:41:23.394805 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:23.394744    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"055998e078e377788acca967b68e12e4c98d34f0daed9c07fb21b1d3ed53c683"} err="failed to get container status \"055998e078e377788acca967b68e12e4c98d34f0daed9c07fb21b1d3ed53c683\": rpc error: code = NotFound desc = could not find container \"055998e078e377788acca967b68e12e4c98d34f0daed9c07fb21b1d3ed53c683\": container with ID starting with 055998e078e377788acca967b68e12e4c98d34f0daed9c07fb21b1d3ed53c683 not found: ID does not exist"
Oct 31 03:41:23.443506 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:41:23.443469    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7168-14a48a\" is forbidden: User \"system:node:ip-10-0-134-255.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-255.us-west-1.compute.internal' and this object" podUID="59ebedaa-2d25-4f2f-98df-22115b365656" pod="opendatahub/test-curl-7168-14a48a"
Oct 31 03:41:24.616808 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:24.616775    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59ebedaa-2d25-4f2f-98df-22115b365656" path="/var/lib/kubelet/pods/59ebedaa-2d25-4f2f-98df-22115b365656/volumes"
Oct 31 03:41:24.656193 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:41:24.656157    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7168-14a48a\" is forbidden: User \"system:node:ip-10-0-134-255.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-255.us-west-1.compute.internal' and this object" podUID="59ebedaa-2d25-4f2f-98df-22115b365656" pod="opendatahub/test-curl-7168-14a48a"
Oct 31 03:41:27.442088 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.442052    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"]
Oct 31 03:41:27.442814 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.442795    4447 cpu_manager.go:414] "containerMap: removing container" podUID="59ebedaa-2d25-4f2f-98df-22115b365656" containerName="test-curl-7168-14a48a"
Oct 31 03:41:27.442853 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.442816    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ebedaa-2d25-4f2f-98df-22115b365656" containerName="test-curl-7168-14a48a"
Oct 31 03:41:27.442993 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.442982    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ebedaa-2d25-4f2f-98df-22115b365656" containerName="test-curl-7168-14a48a"
Oct 31 03:41:27.448358 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.448337    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.509698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.509665    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"]
Oct 31 03:41:27.526079 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.526036    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a1ebb8-9905-470d-bace-caf42a18af34-istiod-ca-cert\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.526079 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.526073    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-istio-envoy\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.526341 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.526091    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92992\" (UniqueName: \"kubernetes.io/projected/01a1ebb8-9905-470d-bace-caf42a18af34-kube-api-access-92992\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.526341 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.526171    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-credential-socket\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.526341 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.526286    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a1ebb8-9905-470d-bace-caf42a18af34-istio-token\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.526341 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.526326    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-istio-data\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.526549 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.526411    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-workload-socket\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.526605 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.526542    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-workload-certs\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.526605 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.526578    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a1ebb8-9905-470d-bace-caf42a18af34-istio-podinfo\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.553740 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.553706    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-909365-openshift-default-dockercfg-rxcsh\""
Oct 31 03:41:27.627771 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.627736    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a1ebb8-9905-470d-bace-caf42a18af34-istio-token\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.627967 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.627806    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-istio-data\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.627967 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.627917    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-workload-socket\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.628102 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.628059    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-workload-certs\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.628159 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.628118    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a1ebb8-9905-470d-bace-caf42a18af34-istio-podinfo\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.628232 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.628186    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a1ebb8-9905-470d-bace-caf42a18af34-istiod-ca-cert\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.628232 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.628213    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-istio-envoy\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.628332 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.628236    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92992\" (UniqueName: \"kubernetes.io/projected/01a1ebb8-9905-470d-bace-caf42a18af34-kube-api-access-92992\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.628332 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.628236    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-istio-data\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.628513 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.628370    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-workload-socket\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.628513 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.628400    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-workload-certs\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.628513 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.628481    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-credential-socket\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.628737 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.628715    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-credential-socket\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.629093 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.629076    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a1ebb8-9905-470d-bace-caf42a18af34-istiod-ca-cert\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.630560 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.630531    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-istio-envoy\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.631133 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.631115    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a1ebb8-9905-470d-bace-caf42a18af34-istio-podinfo\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.690733 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.690702    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a1ebb8-9905-470d-bace-caf42a18af34-istio-token\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.690921 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.690749    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92992\" (UniqueName: \"kubernetes.io/projected/01a1ebb8-9905-470d-bace-caf42a18af34-kube-api-access-92992\") pod \"e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") " pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:27.763147 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:27.763110    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:28.202495 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:41:28.202463    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a1ebb8_9905_470d_bace_caf42a18af34.slice/crio-458f05c3ba098a0b6da22a09450738955c5420cd92b4864ab6a68c6ec863da94 WatchSource:0}: Error finding container 458f05c3ba098a0b6da22a09450738955c5420cd92b4864ab6a68c6ec863da94: Status 404 returned error can't find the container with id 458f05c3ba098a0b6da22a09450738955c5420cd92b4864ab6a68c6ec863da94
Oct 31 03:41:28.202872 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.202847    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"]
Oct 31 03:41:28.205263 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.205230    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:41:28.205384 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.205309    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:41:28.205384 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.205358    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:41:28.430760 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.430662    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld" event={"ID":"01a1ebb8-9905-470d-bace-caf42a18af34","Type":"ContainerStarted","Data":"c394ffc0da9985e3da64bda2a13304450aef3fefe0708c3adb8857cac0b9bcb8"}
Oct 31 03:41:28.430760 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.430728    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld" event={"ID":"01a1ebb8-9905-470d-bace-caf42a18af34","Type":"ContainerStarted","Data":"458f05c3ba098a0b6da22a09450738955c5420cd92b4864ab6a68c6ec863da94"}
Oct 31 03:41:28.506822 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.506791    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"]
Oct 31 03:41:28.511736 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.511714    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:28.541559 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.541522    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/70b3d31c-a1ec-45db-97e5-cf9d1a0f227c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cx854\" (UID: \"70b3d31c-a1ec-45db-97e5-cf9d1a0f227c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:28.541729 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.541603    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/70b3d31c-a1ec-45db-97e5-cf9d1a0f227c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cx854\" (UID: \"70b3d31c-a1ec-45db-97e5-cf9d1a0f227c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:28.541799 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.541785    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b3d31c-a1ec-45db-97e5-cf9d1a0f227c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cx854\" (UID: \"70b3d31c-a1ec-45db-97e5-cf9d1a0f227c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:28.541850 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.541815    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/70b3d31c-a1ec-45db-97e5-cf9d1a0f227c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cx854\" (UID: \"70b3d31c-a1ec-45db-97e5-cf9d1a0f227c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:28.542026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.542006    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqtfx\" (UniqueName: \"kubernetes.io/projected/70b3d31c-a1ec-45db-97e5-cf9d1a0f227c-kube-api-access-pqtfx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cx854\" (UID: \"70b3d31c-a1ec-45db-97e5-cf9d1a0f227c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:28.542090 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.542047    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/70b3d31c-a1ec-45db-97e5-cf9d1a0f227c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cx854\" (UID: \"70b3d31c-a1ec-45db-97e5-cf9d1a0f227c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:28.542242 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.542221    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b3d31c-a1ec-45db-97e5-cf9d1a0f227c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cx854\" (UID: \"70b3d31c-a1ec-45db-97e5-cf9d1a0f227c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:28.564333 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.564282    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld" podStartSLOduration=1.564267785 podStartE2EDuration="1.564267785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:28.563484777 +0000 UTC m=+1314.074467153" watchObservedRunningTime="2026-10-31 03:41:28.564267785 +0000 UTC m=+1314.075250162"
Oct 31 03:41:28.565395 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.565371    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 03:41:28.579510 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.579478    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"]
Oct 31 03:41:28.625350 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.625314    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 03:41:28.625599 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.625442    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-476rz\""
Oct 31 03:41:28.625599 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.625481    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 03:41:28.649952 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.643713    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b3d31c-a1ec-45db-97e5-cf9d1a0f227c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cx854\" (UID: \"70b3d31c-a1ec-45db-97e5-cf9d1a0f227c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:28.649952 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.643828    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/70b3d31c-a1ec-45db-97e5-cf9d1a0f227c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cx854\" (UID: \"70b3d31c-a1ec-45db-97e5-cf9d1a0f227c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:28.649952 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.643935    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/70b3d31c-a1ec-45db-97e5-cf9d1a0f227c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cx854\" (UID: \"70b3d31c-a1ec-45db-97e5-cf9d1a0f227c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:28.649952 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.644025    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b3d31c-a1ec-45db-97e5-cf9d1a0f227c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cx854\" (UID: \"70b3d31c-a1ec-45db-97e5-cf9d1a0f227c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:28.649952 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.644074    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/70b3d31c-a1ec-45db-97e5-cf9d1a0f227c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cx854\" (UID: \"70b3d31c-a1ec-45db-97e5-cf9d1a0f227c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:28.649952 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.644215    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqtfx\" (UniqueName: \"kubernetes.io/projected/70b3d31c-a1ec-45db-97e5-cf9d1a0f227c-kube-api-access-pqtfx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cx854\" (UID: \"70b3d31c-a1ec-45db-97e5-cf9d1a0f227c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:28.649952 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.644253    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/70b3d31c-a1ec-45db-97e5-cf9d1a0f227c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cx854\" (UID: \"70b3d31c-a1ec-45db-97e5-cf9d1a0f227c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:28.649952 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.645085    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/70b3d31c-a1ec-45db-97e5-cf9d1a0f227c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cx854\" (UID: \"70b3d31c-a1ec-45db-97e5-cf9d1a0f227c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:28.651223 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.650929    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/70b3d31c-a1ec-45db-97e5-cf9d1a0f227c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cx854\" (UID: \"70b3d31c-a1ec-45db-97e5-cf9d1a0f227c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:28.651223 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.650946    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/70b3d31c-a1ec-45db-97e5-cf9d1a0f227c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cx854\" (UID: \"70b3d31c-a1ec-45db-97e5-cf9d1a0f227c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:28.651223 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.651187    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b3d31c-a1ec-45db-97e5-cf9d1a0f227c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cx854\" (UID: \"70b3d31c-a1ec-45db-97e5-cf9d1a0f227c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:28.651441 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.651304    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/70b3d31c-a1ec-45db-97e5-cf9d1a0f227c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cx854\" (UID: \"70b3d31c-a1ec-45db-97e5-cf9d1a0f227c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:28.709070 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.707828    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqtfx\" (UniqueName: \"kubernetes.io/projected/70b3d31c-a1ec-45db-97e5-cf9d1a0f227c-kube-api-access-pqtfx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cx854\" (UID: \"70b3d31c-a1ec-45db-97e5-cf9d1a0f227c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:28.709070 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.708507    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b3d31c-a1ec-45db-97e5-cf9d1a0f227c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cx854\" (UID: \"70b3d31c-a1ec-45db-97e5-cf9d1a0f227c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:28.765964 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.763281    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:28.827145 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:28.827115    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:29.308910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:29.308858    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"]
Oct 31 03:41:29.310843 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:41:29.310817    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b3d31c_a1ec_45db_97e5_cf9d1a0f227c.slice/crio-d058688beb7c41f0716438b1cbfdac5c1ba3aa637fbd22eef23eb6678d4917d4 WatchSource:0}: Error finding container d058688beb7c41f0716438b1cbfdac5c1ba3aa637fbd22eef23eb6678d4917d4: Status 404 returned error can't find the container with id d058688beb7c41f0716438b1cbfdac5c1ba3aa637fbd22eef23eb6678d4917d4
Oct 31 03:41:29.440686 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:29.440649    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854" event={"ID":"70b3d31c-a1ec-45db-97e5-cf9d1a0f227c","Type":"ContainerStarted","Data":"d058688beb7c41f0716438b1cbfdac5c1ba3aa637fbd22eef23eb6678d4917d4"}
Oct 31 03:41:29.765855 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:29.765819    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:41:29.766409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:29.765929    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld" podUID="01a1ebb8-9905-470d-bace-caf42a18af34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 31 03:41:30.764008 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:30.763973    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:41:30.764167 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:30.764027    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld" podUID="01a1ebb8-9905-470d-bace-caf42a18af34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 31 03:41:31.764641 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:31.764195    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:41:31.764641 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:31.764273    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld" podUID="01a1ebb8-9905-470d-bace-caf42a18af34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 31 03:41:32.763849 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:32.763814    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:41:32.764083 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:32.763896    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld" podUID="01a1ebb8-9905-470d-bace-caf42a18af34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 31 03:41:33.765172 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:33.764571    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:41:33.765172 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:33.764629    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld" podUID="01a1ebb8-9905-470d-bace-caf42a18af34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 31 03:41:34.109542 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:34.109507    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:41:34.109662 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:34.109583    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:41:34.523971 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:34.523879    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854" event={"ID":"70b3d31c-a1ec-45db-97e5-cf9d1a0f227c","Type":"ContainerStarted","Data":"53c0cdc45826269897fa8b540823745a429c4740640a7b37ef9053dcaf4b9309"}
Oct 31 03:41:34.524179 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:34.524069    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:34.748987 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:34.748923    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854" podStartSLOduration=1.953664399 podStartE2EDuration="6.748906459s" totalImagesPullingTime="4.79524206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:34.694425162 +0000 UTC m=+1320.205407540" watchObservedRunningTime="2026-10-31 03:41:34.748906459 +0000 UTC m=+1320.259888833"
Oct 31 03:41:34.764207 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:34.764179    4447 patch_prober.go:29] interesting pod/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:41:34.764364 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:34.764244    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld" podUID="01a1ebb8-9905-470d-bace-caf42a18af34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 31 03:41:35.552923 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:35.552836    4447 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-cx854 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:41:35.553427 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:35.552929    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854" podUID="70b3d31c-a1ec-45db-97e5-cf9d1a0f227c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:41:35.789528 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:35.788821    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:35.801251 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:35.790506    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:35.801251 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:35.791902    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:41:35.906395 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:35.906304    4447 scope.go:122] "RemoveContainer" containerID="d4944093c2351ad1f06314708a20c6b89f7d5147ae97e38f240024ad6b380acb"
Oct 31 03:41:36.028018 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:36.027984    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw"]
Oct 31 03:41:36.033973 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:36.033950    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw"
Oct 31 03:41:36.086709 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:36.086672    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6w84\" (UniqueName: \"kubernetes.io/projected/668b2bb1-f5a0-454a-bcb5-c28d56aeab40-kube-api-access-k6w84\") pod \"payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw\" (UID: \"668b2bb1-f5a0-454a-bcb5-c28d56aeab40\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw"
Oct 31 03:41:36.086951 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:36.086855    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/668b2bb1-f5a0-454a-bcb5-c28d56aeab40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw\" (UID: \"668b2bb1-f5a0-454a-bcb5-c28d56aeab40\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw"
Oct 31 03:41:36.102437 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:36.102150    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-dafd9df5\""
Oct 31 03:41:36.102437 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:36.102176    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-dafd9df5-dockercfg-lhdpd\""
Oct 31 03:41:36.110062 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:36.110033    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw"]
Oct 31 03:41:36.185377 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:36.185273    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6"]
Oct 31 03:41:36.189944 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:36.189571    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/668b2bb1-f5a0-454a-bcb5-c28d56aeab40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw\" (UID: \"668b2bb1-f5a0-454a-bcb5-c28d56aeab40\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw"
Oct 31 03:41:36.189944 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:36.189930    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6w84\" (UniqueName: \"kubernetes.io/projected/668b2bb1-f5a0-454a-bcb5-c28d56aeab40-kube-api-access-k6w84\") pod \"payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw\" (UID: \"668b2bb1-f5a0-454a-bcb5-c28d56aeab40\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw"
Oct 31 03:41:36.191324 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:36.191300    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/668b2bb1-f5a0-454a-bcb5-c28d56aeab40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw\" (UID: \"668b2bb1-f5a0-454a-bcb5-c28d56aeab40\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw"
Oct 31 03:41:36.196902 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:36.196857    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6"
Oct 31 03:41:36.259953 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:36.255689    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6"]
Oct 31 03:41:36.343177 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:36.343100    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6w84\" (UniqueName: \"kubernetes.io/projected/668b2bb1-f5a0-454a-bcb5-c28d56aeab40-kube-api-access-k6w84\") pod \"payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw\" (UID: \"668b2bb1-f5a0-454a-bcb5-c28d56aeab40\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw"
Oct 31 03:41:36.348619 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:36.348574    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw"
Oct 31 03:41:36.437577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:36.437489    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c0742f-4434-49f0-b2ea-7264667d2933-plugins-config-volume\") pod \"payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6\" (UID: \"d6c0742f-4434-49f0-b2ea-7264667d2933\") " pod="openshift-ingress/payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6"
Oct 31 03:41:36.437769 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:36.437746    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh79c\" (UniqueName: \"kubernetes.io/projected/d6c0742f-4434-49f0-b2ea-7264667d2933-kube-api-access-nh79c\") pod \"payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6\" (UID: \"d6c0742f-4434-49f0-b2ea-7264667d2933\") " pod="openshift-ingress/payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6"
Oct 31 03:41:36.539030 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:36.538979    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh79c\" (UniqueName: \"kubernetes.io/projected/d6c0742f-4434-49f0-b2ea-7264667d2933-kube-api-access-nh79c\") pod \"payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6\" (UID: \"d6c0742f-4434-49f0-b2ea-7264667d2933\") " pod="openshift-ingress/payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6"
Oct 31 03:41:36.539368 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:36.539341    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c0742f-4434-49f0-b2ea-7264667d2933-plugins-config-volume\") pod \"payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6\" (UID: \"d6c0742f-4434-49f0-b2ea-7264667d2933\") " pod="openshift-ingress/payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6"
Oct 31 03:41:36.540026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:36.540004    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c0742f-4434-49f0-b2ea-7264667d2933-plugins-config-volume\") pod \"payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6\" (UID: \"d6c0742f-4434-49f0-b2ea-7264667d2933\") " pod="openshift-ingress/payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6"
Oct 31 03:41:36.600894 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:36.600861    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh79c\" (UniqueName: \"kubernetes.io/projected/d6c0742f-4434-49f0-b2ea-7264667d2933-kube-api-access-nh79c\") pod \"payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6\" (UID: \"d6c0742f-4434-49f0-b2ea-7264667d2933\") " pod="openshift-ingress/payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6"
Oct 31 03:41:36.763900 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:36.763851    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw"]
Oct 31 03:41:36.764033 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:41:36.763998    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod668b2bb1_f5a0_454a_bcb5_c28d56aeab40.slice/crio-5bf97b0a7f9a15f90f366ba94068c45cfa130d12e3f58849c392d4fd027293f0 WatchSource:0}: Error finding container 5bf97b0a7f9a15f90f366ba94068c45cfa130d12e3f58849c392d4fd027293f0: Status 404 returned error can't find the container with id 5bf97b0a7f9a15f90f366ba94068c45cfa130d12e3f58849c392d4fd027293f0
Oct 31 03:41:36.820981 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:36.820957    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6"
Oct 31 03:41:37.207779 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:37.207746    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6"]
Oct 31 03:41:37.208011 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:41:37.207977    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c0742f_4434_49f0_b2ea_7264667d2933.slice/crio-5cee3547a6145a9088afaad7e04f83bd044a24f7d4311b0791a0d5c36c993a03 WatchSource:0}: Error finding container 5cee3547a6145a9088afaad7e04f83bd044a24f7d4311b0791a0d5c36c993a03: Status 404 returned error can't find the container with id 5cee3547a6145a9088afaad7e04f83bd044a24f7d4311b0791a0d5c36c993a03
Oct 31 03:41:37.571429 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:37.571335    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6" event={"ID":"d6c0742f-4434-49f0-b2ea-7264667d2933","Type":"ContainerStarted","Data":"0d641dca117f66756548af85f3f4ef7b7057003df8c840ebb11dde2dc72bcd68"}
Oct 31 03:41:37.571429 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:37.571393    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6" event={"ID":"d6c0742f-4434-49f0-b2ea-7264667d2933","Type":"ContainerStarted","Data":"5cee3547a6145a9088afaad7e04f83bd044a24f7d4311b0791a0d5c36c993a03"}
Oct 31 03:41:37.571653 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:37.571435    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6"
Oct 31 03:41:37.573141 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:37.573115    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw" event={"ID":"668b2bb1-f5a0-454a-bcb5-c28d56aeab40","Type":"ContainerStarted","Data":"98e9248f6c4e9137c17201b6d6226f10003af7b8e2eb07bcbbe311ce3833ce77"}
Oct 31 03:41:37.573247 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:37.573158    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw" event={"ID":"668b2bb1-f5a0-454a-bcb5-c28d56aeab40","Type":"ContainerStarted","Data":"5bf97b0a7f9a15f90f366ba94068c45cfa130d12e3f58849c392d4fd027293f0"}
Oct 31 03:41:37.573247 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:37.573198    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw"
Oct 31 03:41:37.690590 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:37.690540    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6" podStartSLOduration=1.6905267579999999 podStartE2EDuration="1.690526758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:37.687364949 +0000 UTC m=+1323.198347325" watchObservedRunningTime="2026-10-31 03:41:37.690526758 +0000 UTC m=+1323.201509154"
Oct 31 03:41:37.806267 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:37.806200    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw" podStartSLOduration=2.806179858 podStartE2EDuration="2.806179858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:37.804208975 +0000 UTC m=+1323.315191353" watchObservedRunningTime="2026-10-31 03:41:37.806179858 +0000 UTC m=+1323.317162235"
Oct 31 03:41:38.551223 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:38.551196    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cx854"
Oct 31 03:41:41.105068 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:41.105033    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf"]
Oct 31 03:41:41.110274 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:41.110243    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf"
Oct 31 03:41:41.170711 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:41.170673    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf"]
Oct 31 03:41:41.212951 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:41.212916    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-9e38c94c\""
Oct 31 03:41:41.213179 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:41.213156    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-9e38c94c-dockercfg-cwbq7\""
Oct 31 03:41:41.213954 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:41.213932    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5s7q\" (UniqueName: \"kubernetes.io/projected/6e1b9359-2dbe-4916-b124-95f4c793db43-kube-api-access-m5s7q\") pod \"payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf\" (UID: \"6e1b9359-2dbe-4916-b124-95f4c793db43\") " pod="openshift-ingress/payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf"
Oct 31 03:41:41.214085 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:41.214062    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1b9359-2dbe-4916-b124-95f4c793db43-plugins-config-volume\") pod \"payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf\" (UID: \"6e1b9359-2dbe-4916-b124-95f4c793db43\") " pod="openshift-ingress/payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf"
Oct 31 03:41:41.322909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:41.316147    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5s7q\" (UniqueName: \"kubernetes.io/projected/6e1b9359-2dbe-4916-b124-95f4c793db43-kube-api-access-m5s7q\") pod \"payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf\" (UID: \"6e1b9359-2dbe-4916-b124-95f4c793db43\") " pod="openshift-ingress/payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf"
Oct 31 03:41:41.322909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:41.316236    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1b9359-2dbe-4916-b124-95f4c793db43-plugins-config-volume\") pod \"payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf\" (UID: \"6e1b9359-2dbe-4916-b124-95f4c793db43\") " pod="openshift-ingress/payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf"
Oct 31 03:41:41.322909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:41.317229    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1b9359-2dbe-4916-b124-95f4c793db43-plugins-config-volume\") pod \"payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf\" (UID: \"6e1b9359-2dbe-4916-b124-95f4c793db43\") " pod="openshift-ingress/payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf"
Oct 31 03:41:41.381317 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:41.381238    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5s7q\" (UniqueName: \"kubernetes.io/projected/6e1b9359-2dbe-4916-b124-95f4c793db43-kube-api-access-m5s7q\") pod \"payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf\" (UID: \"6e1b9359-2dbe-4916-b124-95f4c793db43\") " pod="openshift-ingress/payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf"
Oct 31 03:41:41.427938 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:41.427545    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf"
Oct 31 03:41:41.810974 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:41:41.810949    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1b9359_2dbe_4916_b124_95f4c793db43.slice/crio-3098e0b71afa56ca204a6532c4c24cc2ace3987cba9db44948a9833fac33faf2 WatchSource:0}: Error finding container 3098e0b71afa56ca204a6532c4c24cc2ace3987cba9db44948a9833fac33faf2: Status 404 returned error can't find the container with id 3098e0b71afa56ca204a6532c4c24cc2ace3987cba9db44948a9833fac33faf2
Oct 31 03:41:41.811379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:41.811359    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf"]
Oct 31 03:41:42.617056 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:42.617022    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf" event={"ID":"6e1b9359-2dbe-4916-b124-95f4c793db43","Type":"ContainerStarted","Data":"34461e998d5bb8a7559556b7bee4f34fa8395ec3942416c1b499c4e2d66e1779"}
Oct 31 03:41:42.617455 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:42.617081    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf" event={"ID":"6e1b9359-2dbe-4916-b124-95f4c793db43","Type":"ContainerStarted","Data":"3098e0b71afa56ca204a6532c4c24cc2ace3987cba9db44948a9833fac33faf2"}
Oct 31 03:41:42.617455 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:42.617103    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf"
Oct 31 03:41:42.770911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:42.769165    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf" podStartSLOduration=1.769138624 podStartE2EDuration="1.769138624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:42.749220855 +0000 UTC m=+1328.260203231" watchObservedRunningTime="2026-10-31 03:41:42.769138624 +0000 UTC m=+1328.280121006"
Oct 31 03:41:45.944206 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:45.944155    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs"]
Oct 31 03:41:45.949122 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:45.949100    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs"
Oct 31 03:41:46.004717 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:46.004405    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9klkm\" (UniqueName: \"kubernetes.io/projected/b1f11045-ca77-415b-893f-88d842338681-kube-api-access-9klkm\") pod \"payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs\" (UID: \"b1f11045-ca77-415b-893f-88d842338681\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs"
Oct 31 03:41:46.004717 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:46.004451    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f11045-ca77-415b-893f-88d842338681-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs\" (UID: \"b1f11045-ca77-415b-893f-88d842338681\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs"
Oct 31 03:41:46.011987 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:46.008396    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs"]
Oct 31 03:41:46.053107 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:46.053078    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-67b1ef6f\""
Oct 31 03:41:46.054853 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:46.053458    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-67b1ef6f-dockercfg-hf9z6\""
Oct 31 03:41:46.156915 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:46.137402    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9klkm\" (UniqueName: \"kubernetes.io/projected/b1f11045-ca77-415b-893f-88d842338681-kube-api-access-9klkm\") pod \"payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs\" (UID: \"b1f11045-ca77-415b-893f-88d842338681\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs"
Oct 31 03:41:46.156915 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:46.137448    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f11045-ca77-415b-893f-88d842338681-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs\" (UID: \"b1f11045-ca77-415b-893f-88d842338681\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs"
Oct 31 03:41:46.156915 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:46.142264    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f11045-ca77-415b-893f-88d842338681-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs\" (UID: \"b1f11045-ca77-415b-893f-88d842338681\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs"
Oct 31 03:41:46.173827 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:46.173774    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq"]
Oct 31 03:41:46.193066 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:46.193036    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq"
Oct 31 03:41:46.199906 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:46.199818    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9klkm\" (UniqueName: \"kubernetes.io/projected/b1f11045-ca77-415b-893f-88d842338681-kube-api-access-9klkm\") pod \"payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs\" (UID: \"b1f11045-ca77-415b-893f-88d842338681\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs"
Oct 31 03:41:46.200630 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:46.200608    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq"]
Oct 31 03:41:46.264156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:46.264129    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs"
Oct 31 03:41:46.343676 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:46.343421    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vpdq\" (UniqueName: \"kubernetes.io/projected/00137945-5937-4cfb-8d79-99dfa03a21e7-kube-api-access-8vpdq\") pod \"payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq\" (UID: \"00137945-5937-4cfb-8d79-99dfa03a21e7\") " pod="openshift-ingress/payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq"
Oct 31 03:41:46.343676 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:46.343492    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00137945-5937-4cfb-8d79-99dfa03a21e7-plugins-config-volume\") pod \"payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq\" (UID: \"00137945-5937-4cfb-8d79-99dfa03a21e7\") " pod="openshift-ingress/payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq"
Oct 31 03:41:46.444575 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:46.444545    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vpdq\" (UniqueName: \"kubernetes.io/projected/00137945-5937-4cfb-8d79-99dfa03a21e7-kube-api-access-8vpdq\") pod \"payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq\" (UID: \"00137945-5937-4cfb-8d79-99dfa03a21e7\") " pod="openshift-ingress/payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq"
Oct 31 03:41:46.444751 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:46.444592    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00137945-5937-4cfb-8d79-99dfa03a21e7-plugins-config-volume\") pod \"payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq\" (UID: \"00137945-5937-4cfb-8d79-99dfa03a21e7\") " pod="openshift-ingress/payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq"
Oct 31 03:41:46.445390 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:46.445370    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00137945-5937-4cfb-8d79-99dfa03a21e7-plugins-config-volume\") pod \"payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq\" (UID: \"00137945-5937-4cfb-8d79-99dfa03a21e7\") " pod="openshift-ingress/payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq"
Oct 31 03:41:46.506250 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:46.506221    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vpdq\" (UniqueName: \"kubernetes.io/projected/00137945-5937-4cfb-8d79-99dfa03a21e7-kube-api-access-8vpdq\") pod \"payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq\" (UID: \"00137945-5937-4cfb-8d79-99dfa03a21e7\") " pod="openshift-ingress/payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq"
Oct 31 03:41:46.533363 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:46.533335    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq"
Oct 31 03:41:46.646924 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:46.646877    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs"]
Oct 31 03:41:46.647525 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:41:46.647493    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f11045_ca77_415b_893f_88d842338681.slice/crio-bb0ae341440aed66fe3429356ab23e0e9431d5e5dfdee48367856ece82e47763 WatchSource:0}: Error finding container bb0ae341440aed66fe3429356ab23e0e9431d5e5dfdee48367856ece82e47763: Status 404 returned error can't find the container with id bb0ae341440aed66fe3429356ab23e0e9431d5e5dfdee48367856ece82e47763
Oct 31 03:41:46.711415 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:46.711384    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs" event={"ID":"b1f11045-ca77-415b-893f-88d842338681","Type":"ContainerStarted","Data":"bb0ae341440aed66fe3429356ab23e0e9431d5e5dfdee48367856ece82e47763"}
Oct 31 03:41:46.926999 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:46.926969    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq"]
Oct 31 03:41:46.927182 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:41:46.927161    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00137945_5937_4cfb_8d79_99dfa03a21e7.slice/crio-9ab4446b9cecf454e8c193e0571676fe4a83f2248040a31c95f96bc2c67b8456 WatchSource:0}: Error finding container 9ab4446b9cecf454e8c193e0571676fe4a83f2248040a31c95f96bc2c67b8456: Status 404 returned error can't find the container with id 9ab4446b9cecf454e8c193e0571676fe4a83f2248040a31c95f96bc2c67b8456
Oct 31 03:41:47.724512 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:47.724466    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq" event={"ID":"00137945-5937-4cfb-8d79-99dfa03a21e7","Type":"ContainerStarted","Data":"0e801ba61080bf03159eb7880fc2170be1ab96d22b5d14128d206dcdd11980c4"}
Oct 31 03:41:47.725124 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:47.724541    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq" event={"ID":"00137945-5937-4cfb-8d79-99dfa03a21e7","Type":"ContainerStarted","Data":"9ab4446b9cecf454e8c193e0571676fe4a83f2248040a31c95f96bc2c67b8456"}
Oct 31 03:41:47.726427 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:47.726341    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq"
Oct 31 03:41:47.727242 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:47.727208    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs" event={"ID":"b1f11045-ca77-415b-893f-88d842338681","Type":"ContainerStarted","Data":"d6421154edead7c187128dc593f01389a4a8d9ea8e63ea86c414bef390f130ab"}
Oct 31 03:41:47.727628 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:47.727602    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs"
Oct 31 03:41:47.851909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:47.851759    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq" podStartSLOduration=1.851722681 podStartE2EDuration="1.851722681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:47.846300936 +0000 UTC m=+1333.357283315" watchObservedRunningTime="2026-10-31 03:41:47.851722681 +0000 UTC m=+1333.362705061"
Oct 31 03:41:47.982524 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:47.982349    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs" podStartSLOduration=2.982327116 podStartE2EDuration="2.982327116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:47.979358442 +0000 UTC m=+1333.490340820" watchObservedRunningTime="2026-10-31 03:41:47.982327116 +0000 UTC m=+1333.493309493"
Oct 31 03:41:48.689941 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:48.687351    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6"
Oct 31 03:41:48.689941 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:48.687472    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw"
Oct 31 03:41:49.211112 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:49.201211    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 31 03:41:49.303312 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:49.303241    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" podUID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:49.483293 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:49.483187    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Oct 31 03:41:49.915970 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:41:49.882158    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:41:50.140912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:50.138035    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:50.168277 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:50.168174    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:50.225114 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:50.223999    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:50.230100 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:50.226028    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:41:50.350015 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:50.318871    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" podUID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:50.350015 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:50.319356    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:41:50.487958 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:50.484112    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:50.487958 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:50.485259    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:41:50.632912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:50.629720    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:41:50.632912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:50.629783    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v" podUID="5f030d4e-f850-4a06-a262-d2060e084bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:51.244372 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:51.244319    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:51.249323 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:51.248723    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 31 03:41:51.269371 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:51.269313    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded"
Oct 31 03:41:51.578855 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:51.578747    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:52.029968 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:51.880987    4447 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/health\": context deadline exceeded"
Oct 31 03:41:52.029968 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:51.881108    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-nbrhn" podUID="98f65062-e7db-4f54-9d76-6addebbebae6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:52.054864 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:41:52.054825    4447 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.429s"
Oct 31 03:41:52.055407 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:52.055384    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:41:52.055616 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:52.055600    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:41:52.250096 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:52.250048    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:52.250951 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:52.250714    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:52.372957 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:52.354638    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" podUID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:41:52.373931 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:52.357089    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:52.727431 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:52.727376    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:41:52.857284 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:52.857252    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:41:52.857474 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:52.857327    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:41:52.857474 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:52.857417    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:41:52.857474 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:52.857449    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:41:52.857652 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:52.857481    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:41:53.628877 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:53.628846    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf"
Oct 31 03:41:58.787052 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:58.787013    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq"
Oct 31 03:41:58.791935 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:41:58.791907    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs"
Oct 31 03:42:04.295627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:04.295569    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6"]
Oct 31 03:42:04.306491 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:04.306460    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6"
Oct 31 03:42:04.364484 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:04.364450    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e9fd0a85-dockercfg-x2pf7\""
Oct 31 03:42:04.364793 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:04.364772    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e9fd0a85\""
Oct 31 03:42:04.366451 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:04.366419    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6"]
Oct 31 03:42:04.445390 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:04.445350    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx8cx\" (UniqueName: \"kubernetes.io/projected/c33afaeb-7d50-4812-99c7-daa2ad29ef13-kube-api-access-gx8cx\") pod \"payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6\" (UID: \"c33afaeb-7d50-4812-99c7-daa2ad29ef13\") " pod="openshift-ingress/payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6"
Oct 31 03:42:04.445599 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:04.445542    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33afaeb-7d50-4812-99c7-daa2ad29ef13-plugins-config-volume\") pod \"payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6\" (UID: \"c33afaeb-7d50-4812-99c7-daa2ad29ef13\") " pod="openshift-ingress/payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6"
Oct 31 03:42:04.546721 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:04.546623    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33afaeb-7d50-4812-99c7-daa2ad29ef13-plugins-config-volume\") pod \"payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6\" (UID: \"c33afaeb-7d50-4812-99c7-daa2ad29ef13\") " pod="openshift-ingress/payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6"
Oct 31 03:42:04.546949 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:04.546931    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx8cx\" (UniqueName: \"kubernetes.io/projected/c33afaeb-7d50-4812-99c7-daa2ad29ef13-kube-api-access-gx8cx\") pod \"payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6\" (UID: \"c33afaeb-7d50-4812-99c7-daa2ad29ef13\") " pod="openshift-ingress/payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6"
Oct 31 03:42:04.547276 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:04.547259    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33afaeb-7d50-4812-99c7-daa2ad29ef13-plugins-config-volume\") pod \"payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6\" (UID: \"c33afaeb-7d50-4812-99c7-daa2ad29ef13\") " pod="openshift-ingress/payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6"
Oct 31 03:42:04.626816 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:04.626782    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx8cx\" (UniqueName: \"kubernetes.io/projected/c33afaeb-7d50-4812-99c7-daa2ad29ef13-kube-api-access-gx8cx\") pod \"payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6\" (UID: \"c33afaeb-7d50-4812-99c7-daa2ad29ef13\") " pod="openshift-ingress/payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6"
Oct 31 03:42:04.627334 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:04.627276    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6"
Oct 31 03:42:05.035046 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:42:05.035010    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33afaeb_7d50_4812_99c7_daa2ad29ef13.slice/crio-d9c180eaa1a48d2f31af2e62ad594c230f38d60d93b6b7ac99fcd8bea942be35 WatchSource:0}: Error finding container d9c180eaa1a48d2f31af2e62ad594c230f38d60d93b6b7ac99fcd8bea942be35: Status 404 returned error can't find the container with id d9c180eaa1a48d2f31af2e62ad594c230f38d60d93b6b7ac99fcd8bea942be35
Oct 31 03:42:05.035370 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:05.035349    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6"]
Oct 31 03:42:05.486273 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:05.486230    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6" event={"ID":"c33afaeb-7d50-4812-99c7-daa2ad29ef13","Type":"ContainerStarted","Data":"be9bbabc11bbe3079fd3e3ebffa5d0e13af009473764dd7ea3087c46cf4d0010"}
Oct 31 03:42:05.486715 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:05.486286    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6" event={"ID":"c33afaeb-7d50-4812-99c7-daa2ad29ef13","Type":"ContainerStarted","Data":"d9c180eaa1a48d2f31af2e62ad594c230f38d60d93b6b7ac99fcd8bea942be35"}
Oct 31 03:42:05.486715 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:05.486460    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6"
Oct 31 03:42:05.603513 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:05.603461    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6" podStartSLOduration=1.603448775 podStartE2EDuration="1.603448775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:05.603035033 +0000 UTC m=+1351.114017409" watchObservedRunningTime="2026-10-31 03:42:05.603448775 +0000 UTC m=+1351.114431151"
Oct 31 03:42:09.317699 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:09.317661    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp"]
Oct 31 03:42:09.322544 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:09.322525    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp"
Oct 31 03:42:09.380284 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:09.380247    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp"]
Oct 31 03:42:09.437909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:09.432151    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-6de37a-dockercfg-b2ggm\""
Oct 31 03:42:09.437909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:09.432220    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-6de37a\""
Oct 31 03:42:09.465867 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:09.449821    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8b3d1c-03b9-4c9a-8ef2-819411125551-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp\" (UID: \"ec8b3d1c-03b9-4c9a-8ef2-819411125551\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp"
Oct 31 03:42:09.510177 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:09.510134    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hczd5\" (UniqueName: \"kubernetes.io/projected/ec8b3d1c-03b9-4c9a-8ef2-819411125551-kube-api-access-hczd5\") pod \"payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp\" (UID: \"ec8b3d1c-03b9-4c9a-8ef2-819411125551\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp"
Oct 31 03:42:09.594774 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:09.594687    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l"]
Oct 31 03:42:09.611665 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:09.611627    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l"]
Oct 31 03:42:09.612237 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:09.612200    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8b3d1c-03b9-4c9a-8ef2-819411125551-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp\" (UID: \"ec8b3d1c-03b9-4c9a-8ef2-819411125551\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp"
Oct 31 03:42:09.612347 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:09.612274    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hczd5\" (UniqueName: \"kubernetes.io/projected/ec8b3d1c-03b9-4c9a-8ef2-819411125551-kube-api-access-hczd5\") pod \"payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp\" (UID: \"ec8b3d1c-03b9-4c9a-8ef2-819411125551\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp"
Oct 31 03:42:09.612839 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:09.612819    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l"
Oct 31 03:42:09.615631 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:09.615463    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8b3d1c-03b9-4c9a-8ef2-819411125551-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp\" (UID: \"ec8b3d1c-03b9-4c9a-8ef2-819411125551\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp"
Oct 31 03:42:09.675845 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:09.675811    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hczd5\" (UniqueName: \"kubernetes.io/projected/ec8b3d1c-03b9-4c9a-8ef2-819411125551-kube-api-access-hczd5\") pod \"payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp\" (UID: \"ec8b3d1c-03b9-4c9a-8ef2-819411125551\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp"
Oct 31 03:42:09.713877 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:09.713845    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/044e191f-f1a6-4e6a-89e5-50da9020417d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l\" (UID: \"044e191f-f1a6-4e6a-89e5-50da9020417d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l"
Oct 31 03:42:09.714281 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:09.714256    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkv47\" (UniqueName: \"kubernetes.io/projected/044e191f-f1a6-4e6a-89e5-50da9020417d-kube-api-access-jkv47\") pod \"payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l\" (UID: \"044e191f-f1a6-4e6a-89e5-50da9020417d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l"
Oct 31 03:42:09.815355 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:09.815313    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkv47\" (UniqueName: \"kubernetes.io/projected/044e191f-f1a6-4e6a-89e5-50da9020417d-kube-api-access-jkv47\") pod \"payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l\" (UID: \"044e191f-f1a6-4e6a-89e5-50da9020417d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l"
Oct 31 03:42:09.815526 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:09.815406    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/044e191f-f1a6-4e6a-89e5-50da9020417d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l\" (UID: \"044e191f-f1a6-4e6a-89e5-50da9020417d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l"
Oct 31 03:42:09.816177 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:09.816160    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/044e191f-f1a6-4e6a-89e5-50da9020417d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l\" (UID: \"044e191f-f1a6-4e6a-89e5-50da9020417d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l"
Oct 31 03:42:09.876773 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:09.876701    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkv47\" (UniqueName: \"kubernetes.io/projected/044e191f-f1a6-4e6a-89e5-50da9020417d-kube-api-access-jkv47\") pod \"payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l\" (UID: \"044e191f-f1a6-4e6a-89e5-50da9020417d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l"
Oct 31 03:42:09.936570 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:09.936542    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp"
Oct 31 03:42:09.957818 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:09.957792    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l"
Oct 31 03:42:10.325492 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:42:10.325464    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8b3d1c_03b9_4c9a_8ef2_819411125551.slice/crio-31e161dd287335115191ddbec079419591940a1a5cc04283654abbba667d6abb WatchSource:0}: Error finding container 31e161dd287335115191ddbec079419591940a1a5cc04283654abbba667d6abb: Status 404 returned error can't find the container with id 31e161dd287335115191ddbec079419591940a1a5cc04283654abbba667d6abb
Oct 31 03:42:10.325819 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:10.325489    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp"]
Oct 31 03:42:10.376146 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:10.376121    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l"]
Oct 31 03:42:10.380429 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:42:10.380405    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044e191f_f1a6_4e6a_89e5_50da9020417d.slice/crio-7f593627dc53512c67f340b84529ad74147289b7061e61a3365367837a2a1e85 WatchSource:0}: Error finding container 7f593627dc53512c67f340b84529ad74147289b7061e61a3365367837a2a1e85: Status 404 returned error can't find the container with id 7f593627dc53512c67f340b84529ad74147289b7061e61a3365367837a2a1e85
Oct 31 03:42:10.592731 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:10.592644    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp" event={"ID":"ec8b3d1c-03b9-4c9a-8ef2-819411125551","Type":"ContainerStarted","Data":"2332b9aa66a169c7811163bc54fe7251b0d67187d78d1896eccfb8ac709cead9"}
Oct 31 03:42:10.592731 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:10.592715    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp"
Oct 31 03:42:10.592982 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:10.592742    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp" event={"ID":"ec8b3d1c-03b9-4c9a-8ef2-819411125551","Type":"ContainerStarted","Data":"31e161dd287335115191ddbec079419591940a1a5cc04283654abbba667d6abb"}
Oct 31 03:42:10.594363 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:10.594335    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l" event={"ID":"044e191f-f1a6-4e6a-89e5-50da9020417d","Type":"ContainerStarted","Data":"29f63dce7e02bbd91cedcf7e82829100f8ca6164abc9efc999353887a350b8a9"}
Oct 31 03:42:10.594503 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:10.594397    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l"
Oct 31 03:42:10.594503 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:10.594420    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l" event={"ID":"044e191f-f1a6-4e6a-89e5-50da9020417d","Type":"ContainerStarted","Data":"7f593627dc53512c67f340b84529ad74147289b7061e61a3365367837a2a1e85"}
Oct 31 03:42:10.709903 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:10.709842    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp" podStartSLOduration=1.709828226 podStartE2EDuration="1.709828226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:10.70856554 +0000 UTC m=+1356.219547918" watchObservedRunningTime="2026-10-31 03:42:10.709828226 +0000 UTC m=+1356.220810602"
Oct 31 03:42:10.829741 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:10.829694    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l" podStartSLOduration=1.829678805 podStartE2EDuration="1.829678805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:10.825075974 +0000 UTC m=+1356.336058354" watchObservedRunningTime="2026-10-31 03:42:10.829678805 +0000 UTC m=+1356.340661181"
Oct 31 03:42:11.575667 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.575635    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"]
Oct 31 03:42:11.580362 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.580338    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.649995 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.649960    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.650198 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.650026    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.650198 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.650166    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa4dc59-0c78-495d-b13a-69545cb0905f-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.650304 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.650214    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.650304 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.650233    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.650377 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.650319    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.650429 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.650387    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.650469 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.650435    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.650552 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.650538    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfqmr\" (UniqueName: \"kubernetes.io/projected/0aa4dc59-0c78-495d-b13a-69545cb0905f-kube-api-access-dfqmr\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.689477 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.688729    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"]
Oct 31 03:42:11.752395 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.752359    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.752823 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.752803    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.753310 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.753096    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa4dc59-0c78-495d-b13a-69545cb0905f-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.753310 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.753158    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.753310 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.753190    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.753310 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.753195    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-credential-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.753310 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.753225    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.753605 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.753335    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.753605 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.753370    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.753605 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.753473    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-data\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.753605 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.753520    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfqmr\" (UniqueName: \"kubernetes.io/projected/0aa4dc59-0c78-495d-b13a-69545cb0905f-kube-api-access-dfqmr\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.753605 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.753544    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-workload-certs\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.753841 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.753745    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa4dc59-0c78-495d-b13a-69545cb0905f-istiod-ca-cert\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.754852 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.754140    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-workload-socket\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.755759 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.755736    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-podinfo\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.756494 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.756472    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-envoy\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.814720 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.814686    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-token\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.814931 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.814780    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfqmr\" (UniqueName: \"kubernetes.io/projected/0aa4dc59-0c78-495d-b13a-69545cb0905f-kube-api-access-dfqmr\") pod \"e2e-worker-w4-096812-openshift-default-656f48f565-slxfd\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") " pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:11.897385 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:11.897289    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:12.336338 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:42:12.336306    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa4dc59_0c78_495d_b13a_69545cb0905f.slice/crio-e9f841bbceb455441a332662966b9fb204f20739f5017864a75132f0a4d0ba4c WatchSource:0}: Error finding container e9f841bbceb455441a332662966b9fb204f20739f5017864a75132f0a4d0ba4c: Status 404 returned error can't find the container with id e9f841bbceb455441a332662966b9fb204f20739f5017864a75132f0a4d0ba4c
Oct 31 03:42:12.336562 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:12.336534    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"]
Oct 31 03:42:12.339273 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:12.339247    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:42:12.339345 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:12.339308    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:42:12.339345 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:12.339338    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:42:12.614478 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:12.614393    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd" event={"ID":"0aa4dc59-0c78-495d-b13a-69545cb0905f","Type":"ContainerStarted","Data":"488e160c5559c3625f830e8ca954913a28923782a1ce79b992fadbc995bf8e8a"}
Oct 31 03:42:12.614478 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:12.614445    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd" event={"ID":"0aa4dc59-0c78-495d-b13a-69545cb0905f","Type":"ContainerStarted","Data":"e9f841bbceb455441a332662966b9fb204f20739f5017864a75132f0a4d0ba4c"}
Oct 31 03:42:12.742944 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:12.742852    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd" podStartSLOduration=1.742833361 podStartE2EDuration="1.742833361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:12.734850383 +0000 UTC m=+1358.245832761" watchObservedRunningTime="2026-10-31 03:42:12.742833361 +0000 UTC m=+1358.253815739"
Oct 31 03:42:12.900970 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:12.900052    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:13.901019 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:13.900983    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 31 03:42:13.901592 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:13.901556    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd" podUID="0aa4dc59-0c78-495d-b13a-69545cb0905f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 31 03:42:14.899332 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:14.899299    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 31 03:42:14.899560 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:14.899359    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd" podUID="0aa4dc59-0c78-495d-b13a-69545cb0905f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 31 03:42:15.898914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:15.897940    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 31 03:42:15.898914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:15.898018    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd" podUID="0aa4dc59-0c78-495d-b13a-69545cb0905f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 31 03:42:16.517953 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:16.516441    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6"
Oct 31 03:42:16.899387 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:16.899294    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 31 03:42:16.899738 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:16.899378    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd" podUID="0aa4dc59-0c78-495d-b13a-69545cb0905f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 31 03:42:17.899953 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:17.898241    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 31 03:42:17.899953 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:17.898317    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd" podUID="0aa4dc59-0c78-495d-b13a-69545cb0905f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 31 03:42:18.897789 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:18.897755    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 31 03:42:18.897999 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:18.897817    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd" podUID="0aa4dc59-0c78-495d-b13a-69545cb0905f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 31 03:42:19.006702 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:42:19.006661    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:42:19.904055 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:19.904028    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:19.904508 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:19.904495    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:19.905354 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:19.905305    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:42:21.605505 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:21.605468    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l"
Oct 31 03:42:21.606039 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:21.605527    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp"
Oct 31 03:42:38.604922 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:38.604010    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:42:38.604922 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:38.604072    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn" podUID="9f503715-aa26-4a05-9391-de0b40a919e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:42:38.757911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:38.754149    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded"
Oct 31 03:42:38.759764 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:38.759730    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" podUID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:42:38.823008 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:38.822964    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:42:38.852910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:38.851040    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:42:39.741908 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:39.739099    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:42:39.770911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:39.755184    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:42:39.770911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:39.755556    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:42:39.785919 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:39.782420    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" podUID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:42:39.820117 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:39.820078    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:42:39.839726 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:39.839674    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:42:39.841280 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:39.841243    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:42:39.846478 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:39.846359    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:42:39.846478 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:39.846409    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz" podUID="d379c75c-04e8-4ab1-8b40-e8de8b0631ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:42:39.911910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:39.901080    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:42:39.930505 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:39.912239    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:42:40.002967 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:39.979298    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:42:40.002967 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:39.979362    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5" podUID="f1e5b912-f46c-4ce3-ad7e-521e506e5e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:42:40.061114 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:40.060408    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:42:40.107944 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:40.088832    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:42:40.273004 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:40.269139    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:42:40.344007 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:40.343976    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 03:42:49.003831 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:42:49.003748    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:42:59.653836 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:59.653802    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"]
Oct 31 03:42:59.658864 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:59.658847    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"
Oct 31 03:42:59.700183 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:59.700149    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6be8a119-cf39-497e-b2c4-c13113f46c16-metrics-certs\") pod \"maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w\" (UID: \"6be8a119-cf39-497e-b2c4-c13113f46c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"
Oct 31 03:42:59.700375 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:59.700328    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be8a119-cf39-497e-b2c4-c13113f46c16-maas-api-tls\") pod \"maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w\" (UID: \"6be8a119-cf39-497e-b2c4-c13113f46c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"
Oct 31 03:42:59.700536 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:59.700516    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be8a119-cf39-497e-b2c4-c13113f46c16-openshift-service-ca\") pod \"maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w\" (UID: \"6be8a119-cf39-497e-b2c4-c13113f46c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"
Oct 31 03:42:59.700616 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:59.700600    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77kd2\" (UniqueName: \"kubernetes.io/projected/6be8a119-cf39-497e-b2c4-c13113f46c16-kube-api-access-77kd2\") pod \"maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w\" (UID: \"6be8a119-cf39-497e-b2c4-c13113f46c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"
Oct 31 03:42:59.761542 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:59.761501    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e9fd0a85-serving-cert\""
Oct 31 03:42:59.766753 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:59.766729    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"]
Oct 31 03:42:59.802540 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:59.802507    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be8a119-cf39-497e-b2c4-c13113f46c16-maas-api-tls\") pod \"maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w\" (UID: \"6be8a119-cf39-497e-b2c4-c13113f46c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"
Oct 31 03:42:59.802950 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:42:59.802818    4447 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-serving-cert: secret "maas-api-e2e-ait-e9fd0a85-serving-cert" not found
Oct 31 03:42:59.803115 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:42:59.803010    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6be8a119-cf39-497e-b2c4-c13113f46c16-maas-api-tls podName:6be8a119-cf39-497e-b2c4-c13113f46c16 nodeName:}" failed. No retries permitted until 2026-10-31 03:43:00.302987669 +0000 UTC m=+1405.813970044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6be8a119-cf39-497e-b2c4-c13113f46c16-maas-api-tls") pod "maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w" (UID: "6be8a119-cf39-497e-b2c4-c13113f46c16") : secret "maas-api-e2e-ait-e9fd0a85-serving-cert" not found
Oct 31 03:42:59.803115 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:59.803038    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be8a119-cf39-497e-b2c4-c13113f46c16-openshift-service-ca\") pod \"maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w\" (UID: \"6be8a119-cf39-497e-b2c4-c13113f46c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"
Oct 31 03:42:59.803115 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:59.803088    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77kd2\" (UniqueName: \"kubernetes.io/projected/6be8a119-cf39-497e-b2c4-c13113f46c16-kube-api-access-77kd2\") pod \"maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w\" (UID: \"6be8a119-cf39-497e-b2c4-c13113f46c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"
Oct 31 03:42:59.803381 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:59.803251    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6be8a119-cf39-497e-b2c4-c13113f46c16-metrics-certs\") pod \"maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w\" (UID: \"6be8a119-cf39-497e-b2c4-c13113f46c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"
Oct 31 03:42:59.804059 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:59.804034    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be8a119-cf39-497e-b2c4-c13113f46c16-openshift-service-ca\") pod \"maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w\" (UID: \"6be8a119-cf39-497e-b2c4-c13113f46c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"
Oct 31 03:42:59.813411 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:59.813384    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6be8a119-cf39-497e-b2c4-c13113f46c16-metrics-certs\") pod \"maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w\" (UID: \"6be8a119-cf39-497e-b2c4-c13113f46c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"
Oct 31 03:42:59.865010 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:42:59.864976    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77kd2\" (UniqueName: \"kubernetes.io/projected/6be8a119-cf39-497e-b2c4-c13113f46c16-kube-api-access-77kd2\") pod \"maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w\" (UID: \"6be8a119-cf39-497e-b2c4-c13113f46c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"
Oct 31 03:43:00.162374 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.162337    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"]
Oct 31 03:43:00.167789 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.167766    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.219789 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.219750    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.220066 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.219821    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.220066 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.219969    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.220066 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.220023    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.220256 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.220067    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.220256 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.220186    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stkfg\" (UniqueName: \"kubernetes.io/projected/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-kube-api-access-stkfg\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.220256 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.220243    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.220411 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.220368    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.220491 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.220466    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.222571 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.222549    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4td2m\""
Oct 31 03:43:00.254963 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.254920    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"]
Oct 31 03:43:00.322293 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.322258    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.322489 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.322304    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.322489 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.322349    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.322489 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.322425    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stkfg\" (UniqueName: \"kubernetes.io/projected/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-kube-api-access-stkfg\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.322489 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.322466    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.322696 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.322570    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.322696 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.322660    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.322802 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.322715    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be8a119-cf39-497e-b2c4-c13113f46c16-maas-api-tls\") pod \"maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w\" (UID: \"6be8a119-cf39-497e-b2c4-c13113f46c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"
Oct 31 03:43:00.322802 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.322729    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.323033 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.322801    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.323033 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.322845    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.323033 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.322739    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.323191 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.323098    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.323517 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.323477    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.323774 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.323749    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.326034 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.325989    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.326034 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.326024    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be8a119-cf39-497e-b2c4-c13113f46c16-maas-api-tls\") pod \"maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w\" (UID: \"6be8a119-cf39-497e-b2c4-c13113f46c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"
Oct 31 03:43:00.326198 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.326007    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.383675 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.383639    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.384083 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.384059    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stkfg\" (UniqueName: \"kubernetes.io/projected/92da6b1c-2d25-4e59-b877-a9b3b265ab3f-kube-api-access-stkfg\") pod \"maas-default-gateway-openshift-default-fd4b99879-jgmq4\" (UID: \"92da6b1c-2d25-4e59-b877-a9b3b265ab3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.485354 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.485324    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:00.574452 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.574201    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"
Oct 31 03:43:00.919635 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:43:00.919606    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92da6b1c_2d25_4e59_b877_a9b3b265ab3f.slice/crio-1ac17ca6629acf3460f1df361fd34112e3b6694173f4914a1fdb73d46b50a1a2 WatchSource:0}: Error finding container 1ac17ca6629acf3460f1df361fd34112e3b6694173f4914a1fdb73d46b50a1a2: Status 404 returned error can't find the container with id 1ac17ca6629acf3460f1df361fd34112e3b6694173f4914a1fdb73d46b50a1a2
Oct 31 03:43:00.920023 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.920003    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"]
Oct 31 03:43:00.922806 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.922776    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:43:00.922972 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.922839    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:43:00.922972 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:00.922866    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:43:01.018545 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:43:01.018502    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be8a119_cf39_497e_b2c4_c13113f46c16.slice/crio-63278f6651c608f9f95cb153e9fc40699bf869061a76eea8e0b9a78de4e7e984 WatchSource:0}: Error finding container 63278f6651c608f9f95cb153e9fc40699bf869061a76eea8e0b9a78de4e7e984: Status 404 returned error can't find the container with id 63278f6651c608f9f95cb153e9fc40699bf869061a76eea8e0b9a78de4e7e984
Oct 31 03:43:01.018671 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:01.018653    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"]
Oct 31 03:43:01.233904 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:01.233853    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w" event={"ID":"6be8a119-cf39-497e-b2c4-c13113f46c16","Type":"ContainerStarted","Data":"63278f6651c608f9f95cb153e9fc40699bf869061a76eea8e0b9a78de4e7e984"}
Oct 31 03:43:01.236325 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:01.236293    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4" event={"ID":"92da6b1c-2d25-4e59-b877-a9b3b265ab3f","Type":"ContainerStarted","Data":"50f76bf598c2240136b240c6cd1086068ccaf38535aa80f1693dcceec58e4451"}
Oct 31 03:43:01.236443 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:01.236348    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4" event={"ID":"92da6b1c-2d25-4e59-b877-a9b3b265ab3f","Type":"ContainerStarted","Data":"1ac17ca6629acf3460f1df361fd34112e3b6694173f4914a1fdb73d46b50a1a2"}
Oct 31 03:43:01.363755 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:01.363695    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4" podStartSLOduration=1.3636821829999999 podStartE2EDuration="1.363682183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:01.35823196 +0000 UTC m=+1406.869214337" watchObservedRunningTime="2026-10-31 03:43:01.363682183 +0000 UTC m=+1406.874664560"
Oct 31 03:43:01.497976 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:01.489097    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:02.246373 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:02.246339    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w" event={"ID":"6be8a119-cf39-497e-b2c4-c13113f46c16","Type":"ContainerStarted","Data":"aec89b3a1d59368f6403270a864879c31a60b2ed6b969abf1d330d14b29fdd8a"}
Oct 31 03:43:02.246790 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:02.246438    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"
Oct 31 03:43:02.367664 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:02.367607    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w" podStartSLOduration=2.539526591 podStartE2EDuration="3.367591302s" totalImagesPullingTime="828.064711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:02.364155901 +0000 UTC m=+1407.875138280" watchObservedRunningTime="2026-10-31 03:43:02.367591302 +0000 UTC m=+1407.878573678"
Oct 31 03:43:02.497004 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:02.489391    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jgmq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 31 03:43:02.497004 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:02.489475    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4" podUID="92da6b1c-2d25-4e59-b877-a9b3b265ab3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 31 03:43:03.496911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:03.494274    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jgmq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 31 03:43:03.496911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:03.494336    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4" podUID="92da6b1c-2d25-4e59-b877-a9b3b265ab3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 31 03:43:04.496877 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:04.496842    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jgmq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 31 03:43:04.497171 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:04.497138    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4" podUID="92da6b1c-2d25-4e59-b877-a9b3b265ab3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 31 03:43:05.495587 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:05.495546    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jgmq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 31 03:43:05.495896 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:05.495852    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4" podUID="92da6b1c-2d25-4e59-b877-a9b3b265ab3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 31 03:43:06.500344 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:06.500309    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jgmq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 31 03:43:06.500958 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:06.500926    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4" podUID="92da6b1c-2d25-4e59-b877-a9b3b265ab3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 31 03:43:07.486211 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:07.486171    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jgmq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 31 03:43:07.486386 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:07.486238    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4" podUID="92da6b1c-2d25-4e59-b877-a9b3b265ab3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 31 03:43:08.261978 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:08.261951    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"
Oct 31 03:43:09.153081 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:09.153041    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:09.154551 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:09.154522    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:09.155294 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:09.155272    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4"
Oct 31 03:43:19.074349 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:43:19.074308    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:43:37.574644 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:37.574607    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6"]
Oct 31 03:43:37.683352 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:37.683316    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw"]
Oct 31 03:43:42.575605 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:42.575568    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6" podUID="d6c0742f-4434-49f0-b2ea-7264667d2933" containerName="payload-processing" containerID="cri-o://0d641dca117f66756548af85f3f4ef7b7057003df8c840ebb11dde2dc72bcd68" gracePeriod=25
Oct 31 03:43:42.600552 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:42.600524    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf"]
Oct 31 03:43:42.684490 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:42.684449    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw" podUID="668b2bb1-f5a0-454a-bcb5-c28d56aeab40" containerName="payload-pre-processing" containerID="cri-o://98e9248f6c4e9137c17201b6d6226f10003af7b8e2eb07bcbbe311ce3833ce77" gracePeriod=25
Oct 31 03:43:42.916898 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:42.916847    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6"
Oct 31 03:43:42.980771 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:42.980733    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c0742f-4434-49f0-b2ea-7264667d2933-kube-api-access-nh79c\" (UniqueName: \"kubernetes.io/projected/d6c0742f-4434-49f0-b2ea-7264667d2933-kube-api-access-nh79c\") pod \"d6c0742f-4434-49f0-b2ea-7264667d2933\" (UID: \"d6c0742f-4434-49f0-b2ea-7264667d2933\") "
Oct 31 03:43:42.980975 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:42.980876    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c0742f-4434-49f0-b2ea-7264667d2933-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c0742f-4434-49f0-b2ea-7264667d2933-plugins-config-volume\") pod \"d6c0742f-4434-49f0-b2ea-7264667d2933\" (UID: \"d6c0742f-4434-49f0-b2ea-7264667d2933\") "
Oct 31 03:43:42.982265 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:42.982220    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c0742f-4434-49f0-b2ea-7264667d2933-plugins-config-volume" pod "d6c0742f-4434-49f0-b2ea-7264667d2933" (UID: "d6c0742f-4434-49f0-b2ea-7264667d2933"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:43:42.989420 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:42.989394    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c0742f-4434-49f0-b2ea-7264667d2933-kube-api-access-nh79c" pod "d6c0742f-4434-49f0-b2ea-7264667d2933" (UID: "d6c0742f-4434-49f0-b2ea-7264667d2933"). InnerVolumeSpecName "kube-api-access-nh79c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:43.014854 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.014811    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw"
Oct 31 03:43:43.092490 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.092457    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/668b2bb1-f5a0-454a-bcb5-c28d56aeab40-kube-api-access-k6w84\" (UniqueName: \"kubernetes.io/projected/668b2bb1-f5a0-454a-bcb5-c28d56aeab40-kube-api-access-k6w84\") pod \"668b2bb1-f5a0-454a-bcb5-c28d56aeab40\" (UID: \"668b2bb1-f5a0-454a-bcb5-c28d56aeab40\") "
Oct 31 03:43:43.093131 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.093105    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/668b2bb1-f5a0-454a-bcb5-c28d56aeab40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/668b2bb1-f5a0-454a-bcb5-c28d56aeab40-plugins-config-volume\") pod \"668b2bb1-f5a0-454a-bcb5-c28d56aeab40\" (UID: \"668b2bb1-f5a0-454a-bcb5-c28d56aeab40\") "
Oct 31 03:43:43.093561 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.093535    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/668b2bb1-f5a0-454a-bcb5-c28d56aeab40-plugins-config-volume" pod "668b2bb1-f5a0-454a-bcb5-c28d56aeab40" (UID: "668b2bb1-f5a0-454a-bcb5-c28d56aeab40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:43:43.094619 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.094593    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c0742f-4434-49f0-b2ea-7264667d2933-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:43:43.094723 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.094621    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh79c\" (UniqueName: \"kubernetes.io/projected/d6c0742f-4434-49f0-b2ea-7264667d2933-kube-api-access-nh79c\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:43:43.094723 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.094636    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/668b2bb1-f5a0-454a-bcb5-c28d56aeab40-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:43:43.098133 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.098103    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668b2bb1-f5a0-454a-bcb5-c28d56aeab40-kube-api-access-k6w84" pod "668b2bb1-f5a0-454a-bcb5-c28d56aeab40" (UID: "668b2bb1-f5a0-454a-bcb5-c28d56aeab40"). InnerVolumeSpecName "kube-api-access-k6w84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:43.196645 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.196547    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6w84\" (UniqueName: \"kubernetes.io/projected/668b2bb1-f5a0-454a-bcb5-c28d56aeab40-kube-api-access-k6w84\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:43:43.652673 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.652640    4447 generic.go:356] "Generic (PLEG): container finished" podID="d6c0742f-4434-49f0-b2ea-7264667d2933" containerID="0d641dca117f66756548af85f3f4ef7b7057003df8c840ebb11dde2dc72bcd68" exitCode=0
Oct 31 03:43:43.653190 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.652699    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6"
Oct 31 03:43:43.653190 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.652740    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6" event={"ID":"d6c0742f-4434-49f0-b2ea-7264667d2933","Type":"ContainerDied","Data":"0d641dca117f66756548af85f3f4ef7b7057003df8c840ebb11dde2dc72bcd68"}
Oct 31 03:43:43.653190 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.652807    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6" event={"ID":"d6c0742f-4434-49f0-b2ea-7264667d2933","Type":"ContainerDied","Data":"5cee3547a6145a9088afaad7e04f83bd044a24f7d4311b0791a0d5c36c993a03"}
Oct 31 03:43:43.653190 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.652835    4447 scope.go:122] "RemoveContainer" containerID="0d641dca117f66756548af85f3f4ef7b7057003df8c840ebb11dde2dc72bcd68"
Oct 31 03:43:43.654329 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.654300    4447 generic.go:356] "Generic (PLEG): container finished" podID="668b2bb1-f5a0-454a-bcb5-c28d56aeab40" containerID="98e9248f6c4e9137c17201b6d6226f10003af7b8e2eb07bcbbe311ce3833ce77" exitCode=0
Oct 31 03:43:43.654446 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.654346    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw"
Oct 31 03:43:43.654629 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.654604    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw" event={"ID":"668b2bb1-f5a0-454a-bcb5-c28d56aeab40","Type":"ContainerDied","Data":"98e9248f6c4e9137c17201b6d6226f10003af7b8e2eb07bcbbe311ce3833ce77"}
Oct 31 03:43:43.654683 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.654666    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw" event={"ID":"668b2bb1-f5a0-454a-bcb5-c28d56aeab40","Type":"ContainerDied","Data":"5bf97b0a7f9a15f90f366ba94068c45cfa130d12e3f58849c392d4fd027293f0"}
Oct 31 03:43:43.666175 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.666013    4447 scope.go:122] "RemoveContainer" containerID="0d641dca117f66756548af85f3f4ef7b7057003df8c840ebb11dde2dc72bcd68"
Oct 31 03:43:43.666477 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:43:43.666449    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d641dca117f66756548af85f3f4ef7b7057003df8c840ebb11dde2dc72bcd68\": container with ID starting with 0d641dca117f66756548af85f3f4ef7b7057003df8c840ebb11dde2dc72bcd68 not found: ID does not exist" containerID="0d641dca117f66756548af85f3f4ef7b7057003df8c840ebb11dde2dc72bcd68"
Oct 31 03:43:43.666647 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.666616    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d641dca117f66756548af85f3f4ef7b7057003df8c840ebb11dde2dc72bcd68"} err="failed to get container status \"0d641dca117f66756548af85f3f4ef7b7057003df8c840ebb11dde2dc72bcd68\": rpc error: code = NotFound desc = could not find container \"0d641dca117f66756548af85f3f4ef7b7057003df8c840ebb11dde2dc72bcd68\": container with ID starting with 0d641dca117f66756548af85f3f4ef7b7057003df8c840ebb11dde2dc72bcd68 not found: ID does not exist"
Oct 31 03:43:43.666749 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.666738    4447 scope.go:122] "RemoveContainer" containerID="98e9248f6c4e9137c17201b6d6226f10003af7b8e2eb07bcbbe311ce3833ce77"
Oct 31 03:43:43.683209 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.683176    4447 scope.go:122] "RemoveContainer" containerID="98e9248f6c4e9137c17201b6d6226f10003af7b8e2eb07bcbbe311ce3833ce77"
Oct 31 03:43:43.683572 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:43:43.683541    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98e9248f6c4e9137c17201b6d6226f10003af7b8e2eb07bcbbe311ce3833ce77\": container with ID starting with 98e9248f6c4e9137c17201b6d6226f10003af7b8e2eb07bcbbe311ce3833ce77 not found: ID does not exist" containerID="98e9248f6c4e9137c17201b6d6226f10003af7b8e2eb07bcbbe311ce3833ce77"
Oct 31 03:43:43.683660 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.683586    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98e9248f6c4e9137c17201b6d6226f10003af7b8e2eb07bcbbe311ce3833ce77"} err="failed to get container status \"98e9248f6c4e9137c17201b6d6226f10003af7b8e2eb07bcbbe311ce3833ce77\": rpc error: code = NotFound desc = could not find container \"98e9248f6c4e9137c17201b6d6226f10003af7b8e2eb07bcbbe311ce3833ce77\": container with ID starting with 98e9248f6c4e9137c17201b6d6226f10003af7b8e2eb07bcbbe311ce3833ce77 not found: ID does not exist"
Oct 31 03:43:43.821591 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.821550    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6"]
Oct 31 03:43:43.826438 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.826408    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dafd9df5-64bd7b69c-vwqf6"]
Oct 31 03:43:43.934556 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.934468    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw"]
Oct 31 03:43:43.940787 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:43.940760    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dafd9df5-8799dbb5f-k2kmw"]
Oct 31 03:43:44.613872 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:44.613829    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="668b2bb1-f5a0-454a-bcb5-c28d56aeab40" path="/var/lib/kubelet/pods/668b2bb1-f5a0-454a-bcb5-c28d56aeab40/volumes"
Oct 31 03:43:44.614519 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:44.614496    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6c0742f-4434-49f0-b2ea-7264667d2933" path="/var/lib/kubelet/pods/d6c0742f-4434-49f0-b2ea-7264667d2933/volumes"
Oct 31 03:43:47.601170 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:47.601111    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf" podUID="6e1b9359-2dbe-4916-b124-95f4c793db43" containerName="payload-processing" containerID="cri-o://34461e998d5bb8a7559556b7bee4f34fa8395ec3942416c1b499c4e2d66e1779" gracePeriod=25
Oct 31 03:43:47.884321 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:47.884294    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf"
Oct 31 03:43:47.974758 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:47.974725    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e1b9359-2dbe-4916-b124-95f4c793db43-kube-api-access-m5s7q\" (UniqueName: \"kubernetes.io/projected/6e1b9359-2dbe-4916-b124-95f4c793db43-kube-api-access-m5s7q\") pod \"6e1b9359-2dbe-4916-b124-95f4c793db43\" (UID: \"6e1b9359-2dbe-4916-b124-95f4c793db43\") "
Oct 31 03:43:47.975063 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:47.975039    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e1b9359-2dbe-4916-b124-95f4c793db43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1b9359-2dbe-4916-b124-95f4c793db43-plugins-config-volume\") pod \"6e1b9359-2dbe-4916-b124-95f4c793db43\" (UID: \"6e1b9359-2dbe-4916-b124-95f4c793db43\") "
Oct 31 03:43:47.975406 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:47.975385    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e1b9359-2dbe-4916-b124-95f4c793db43-plugins-config-volume" pod "6e1b9359-2dbe-4916-b124-95f4c793db43" (UID: "6e1b9359-2dbe-4916-b124-95f4c793db43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:43:47.975674 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:47.975655    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1b9359-2dbe-4916-b124-95f4c793db43-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:43:47.977044 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:47.977022    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1b9359-2dbe-4916-b124-95f4c793db43-kube-api-access-m5s7q" pod "6e1b9359-2dbe-4916-b124-95f4c793db43" (UID: "6e1b9359-2dbe-4916-b124-95f4c793db43"). InnerVolumeSpecName "kube-api-access-m5s7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:48.077173 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:48.077139    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5s7q\" (UniqueName: \"kubernetes.io/projected/6e1b9359-2dbe-4916-b124-95f4c793db43-kube-api-access-m5s7q\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:43:48.705337 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:48.705290    4447 generic.go:356] "Generic (PLEG): container finished" podID="6e1b9359-2dbe-4916-b124-95f4c793db43" containerID="34461e998d5bb8a7559556b7bee4f34fa8395ec3942416c1b499c4e2d66e1779" exitCode=0
Oct 31 03:43:48.705337 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:48.705338    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf"
Oct 31 03:43:48.705847 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:48.705374    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf" event={"ID":"6e1b9359-2dbe-4916-b124-95f4c793db43","Type":"ContainerDied","Data":"34461e998d5bb8a7559556b7bee4f34fa8395ec3942416c1b499c4e2d66e1779"}
Oct 31 03:43:48.705847 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:48.705427    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf" event={"ID":"6e1b9359-2dbe-4916-b124-95f4c793db43","Type":"ContainerDied","Data":"3098e0b71afa56ca204a6532c4c24cc2ace3987cba9db44948a9833fac33faf2"}
Oct 31 03:43:48.705847 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:48.705451    4447 scope.go:122] "RemoveContainer" containerID="34461e998d5bb8a7559556b7bee4f34fa8395ec3942416c1b499c4e2d66e1779"
Oct 31 03:43:48.716139 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:48.716114    4447 scope.go:122] "RemoveContainer" containerID="34461e998d5bb8a7559556b7bee4f34fa8395ec3942416c1b499c4e2d66e1779"
Oct 31 03:43:48.716417 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:43:48.716396    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34461e998d5bb8a7559556b7bee4f34fa8395ec3942416c1b499c4e2d66e1779\": container with ID starting with 34461e998d5bb8a7559556b7bee4f34fa8395ec3942416c1b499c4e2d66e1779 not found: ID does not exist" containerID="34461e998d5bb8a7559556b7bee4f34fa8395ec3942416c1b499c4e2d66e1779"
Oct 31 03:43:48.716471 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:48.716426    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34461e998d5bb8a7559556b7bee4f34fa8395ec3942416c1b499c4e2d66e1779"} err="failed to get container status \"34461e998d5bb8a7559556b7bee4f34fa8395ec3942416c1b499c4e2d66e1779\": rpc error: code = NotFound desc = could not find container \"34461e998d5bb8a7559556b7bee4f34fa8395ec3942416c1b499c4e2d66e1779\": container with ID starting with 34461e998d5bb8a7559556b7bee4f34fa8395ec3942416c1b499c4e2d66e1779 not found: ID does not exist"
Oct 31 03:43:48.875236 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:48.875156    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf"]
Oct 31 03:43:48.924959 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:48.924925    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9e38c94c-78759c48f9-95cnf"]
Oct 31 03:43:49.003412 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:43:49.003367    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:43:50.611481 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:50.611449    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e1b9359-2dbe-4916-b124-95f4c793db43" path="/var/lib/kubelet/pods/6e1b9359-2dbe-4916-b124-95f4c793db43/volumes"
Oct 31 03:43:54.217294 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:54.217257    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq"]
Oct 31 03:43:54.422672 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:54.422639    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs"]
Oct 31 03:43:59.218454 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.218407    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq" podUID="00137945-5937-4cfb-8d79-99dfa03a21e7" containerName="payload-processing" containerID="cri-o://0e801ba61080bf03159eb7880fc2170be1ab96d22b5d14128d206dcdd11980c4" gracePeriod=25
Oct 31 03:43:59.423200 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.423153    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs" podUID="b1f11045-ca77-415b-893f-88d842338681" containerName="payload-pre-processing" containerID="cri-o://d6421154edead7c187128dc593f01389a4a8d9ea8e63ea86c414bef390f130ab" gracePeriod=25
Oct 31 03:43:59.567569 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.567533    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq"
Oct 31 03:43:59.702765 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.702737    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs"
Oct 31 03:43:59.745134 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.745102    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00137945-5937-4cfb-8d79-99dfa03a21e7-kube-api-access-8vpdq\" (UniqueName: \"kubernetes.io/projected/00137945-5937-4cfb-8d79-99dfa03a21e7-kube-api-access-8vpdq\") pod \"00137945-5937-4cfb-8d79-99dfa03a21e7\" (UID: \"00137945-5937-4cfb-8d79-99dfa03a21e7\") "
Oct 31 03:43:59.745347 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.745325    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00137945-5937-4cfb-8d79-99dfa03a21e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00137945-5937-4cfb-8d79-99dfa03a21e7-plugins-config-volume\") pod \"00137945-5937-4cfb-8d79-99dfa03a21e7\" (UID: \"00137945-5937-4cfb-8d79-99dfa03a21e7\") "
Oct 31 03:43:59.745677 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.745652    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00137945-5937-4cfb-8d79-99dfa03a21e7-plugins-config-volume" pod "00137945-5937-4cfb-8d79-99dfa03a21e7" (UID: "00137945-5937-4cfb-8d79-99dfa03a21e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:43:59.746119 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.746092    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00137945-5937-4cfb-8d79-99dfa03a21e7-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:43:59.747633 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.747609    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00137945-5937-4cfb-8d79-99dfa03a21e7-kube-api-access-8vpdq" pod "00137945-5937-4cfb-8d79-99dfa03a21e7" (UID: "00137945-5937-4cfb-8d79-99dfa03a21e7"). InnerVolumeSpecName "kube-api-access-8vpdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:59.847431 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.847323    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1f11045-ca77-415b-893f-88d842338681-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f11045-ca77-415b-893f-88d842338681-plugins-config-volume\") pod \"b1f11045-ca77-415b-893f-88d842338681\" (UID: \"b1f11045-ca77-415b-893f-88d842338681\") "
Oct 31 03:43:59.847637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.847569    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1f11045-ca77-415b-893f-88d842338681-kube-api-access-9klkm\" (UniqueName: \"kubernetes.io/projected/b1f11045-ca77-415b-893f-88d842338681-kube-api-access-9klkm\") pod \"b1f11045-ca77-415b-893f-88d842338681\" (UID: \"b1f11045-ca77-415b-893f-88d842338681\") "
Oct 31 03:43:59.847823 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.847794    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1f11045-ca77-415b-893f-88d842338681-plugins-config-volume" pod "b1f11045-ca77-415b-893f-88d842338681" (UID: "b1f11045-ca77-415b-893f-88d842338681"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:43:59.848245 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.848227    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f11045-ca77-415b-893f-88d842338681-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:43:59.848245 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.848246    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vpdq\" (UniqueName: \"kubernetes.io/projected/00137945-5937-4cfb-8d79-99dfa03a21e7-kube-api-access-8vpdq\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:43:59.850568 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.850538    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f11045-ca77-415b-893f-88d842338681-kube-api-access-9klkm" pod "b1f11045-ca77-415b-893f-88d842338681" (UID: "b1f11045-ca77-415b-893f-88d842338681"). InnerVolumeSpecName "kube-api-access-9klkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:59.890609 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.890573    4447 generic.go:356] "Generic (PLEG): container finished" podID="00137945-5937-4cfb-8d79-99dfa03a21e7" containerID="0e801ba61080bf03159eb7880fc2170be1ab96d22b5d14128d206dcdd11980c4" exitCode=0
Oct 31 03:43:59.890774 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.890628    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq"
Oct 31 03:43:59.890774 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.890658    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq" event={"ID":"00137945-5937-4cfb-8d79-99dfa03a21e7","Type":"ContainerDied","Data":"0e801ba61080bf03159eb7880fc2170be1ab96d22b5d14128d206dcdd11980c4"}
Oct 31 03:43:59.890774 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.890715    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq" event={"ID":"00137945-5937-4cfb-8d79-99dfa03a21e7","Type":"ContainerDied","Data":"9ab4446b9cecf454e8c193e0571676fe4a83f2248040a31c95f96bc2c67b8456"}
Oct 31 03:43:59.890774 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.890739    4447 scope.go:122] "RemoveContainer" containerID="0e801ba61080bf03159eb7880fc2170be1ab96d22b5d14128d206dcdd11980c4"
Oct 31 03:43:59.892180 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.892158    4447 generic.go:356] "Generic (PLEG): container finished" podID="b1f11045-ca77-415b-893f-88d842338681" containerID="d6421154edead7c187128dc593f01389a4a8d9ea8e63ea86c414bef390f130ab" exitCode=0
Oct 31 03:43:59.892297 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.892201    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs"
Oct 31 03:43:59.892297 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.892223    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs" event={"ID":"b1f11045-ca77-415b-893f-88d842338681","Type":"ContainerDied","Data":"d6421154edead7c187128dc593f01389a4a8d9ea8e63ea86c414bef390f130ab"}
Oct 31 03:43:59.892297 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.892262    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs" event={"ID":"b1f11045-ca77-415b-893f-88d842338681","Type":"ContainerDied","Data":"bb0ae341440aed66fe3429356ab23e0e9431d5e5dfdee48367856ece82e47763"}
Oct 31 03:43:59.903227 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.903172    4447 scope.go:122] "RemoveContainer" containerID="0e801ba61080bf03159eb7880fc2170be1ab96d22b5d14128d206dcdd11980c4"
Oct 31 03:43:59.903521 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:43:59.903501    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e801ba61080bf03159eb7880fc2170be1ab96d22b5d14128d206dcdd11980c4\": container with ID starting with 0e801ba61080bf03159eb7880fc2170be1ab96d22b5d14128d206dcdd11980c4 not found: ID does not exist" containerID="0e801ba61080bf03159eb7880fc2170be1ab96d22b5d14128d206dcdd11980c4"
Oct 31 03:43:59.903564 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.903530    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e801ba61080bf03159eb7880fc2170be1ab96d22b5d14128d206dcdd11980c4"} err="failed to get container status \"0e801ba61080bf03159eb7880fc2170be1ab96d22b5d14128d206dcdd11980c4\": rpc error: code = NotFound desc = could not find container \"0e801ba61080bf03159eb7880fc2170be1ab96d22b5d14128d206dcdd11980c4\": container with ID starting with 0e801ba61080bf03159eb7880fc2170be1ab96d22b5d14128d206dcdd11980c4 not found: ID does not exist"
Oct 31 03:43:59.903564 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.903548    4447 scope.go:122] "RemoveContainer" containerID="d6421154edead7c187128dc593f01389a4a8d9ea8e63ea86c414bef390f130ab"
Oct 31 03:43:59.914300 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.914284    4447 scope.go:122] "RemoveContainer" containerID="d6421154edead7c187128dc593f01389a4a8d9ea8e63ea86c414bef390f130ab"
Oct 31 03:43:59.914581 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:43:59.914562    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6421154edead7c187128dc593f01389a4a8d9ea8e63ea86c414bef390f130ab\": container with ID starting with d6421154edead7c187128dc593f01389a4a8d9ea8e63ea86c414bef390f130ab not found: ID does not exist" containerID="d6421154edead7c187128dc593f01389a4a8d9ea8e63ea86c414bef390f130ab"
Oct 31 03:43:59.914650 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.914590    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6421154edead7c187128dc593f01389a4a8d9ea8e63ea86c414bef390f130ab"} err="failed to get container status \"d6421154edead7c187128dc593f01389a4a8d9ea8e63ea86c414bef390f130ab\": rpc error: code = NotFound desc = could not find container \"d6421154edead7c187128dc593f01389a4a8d9ea8e63ea86c414bef390f130ab\": container with ID starting with d6421154edead7c187128dc593f01389a4a8d9ea8e63ea86c414bef390f130ab not found: ID does not exist"
Oct 31 03:43:59.949423 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:43:59.949391    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9klkm\" (UniqueName: \"kubernetes.io/projected/b1f11045-ca77-415b-893f-88d842338681-kube-api-access-9klkm\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:44:00.059916 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:00.059825    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs"]
Oct 31 03:44:00.065462 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:00.065433    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67b1ef6f-5c64df7cf-vdkvs"]
Oct 31 03:44:00.178041 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:00.177959    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq"]
Oct 31 03:44:00.183064 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:00.183013    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67b1ef6f-54f5f86497-fl9bq"]
Oct 31 03:44:00.610275 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:00.610240    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00137945-5937-4cfb-8d79-99dfa03a21e7" path="/var/lib/kubelet/pods/00137945-5937-4cfb-8d79-99dfa03a21e7/volumes"
Oct 31 03:44:00.610790 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:00.610773    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1f11045-ca77-415b-893f-88d842338681" path="/var/lib/kubelet/pods/b1f11045-ca77-415b-893f-88d842338681/volumes"
Oct 31 03:44:19.004022 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:44:19.003940    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:44:25.109431 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:25.109397    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6"]
Oct 31 03:44:30.109934 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:30.109824    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6" podUID="c33afaeb-7d50-4812-99c7-daa2ad29ef13" containerName="payload-processing" containerID="cri-o://be9bbabc11bbe3079fd3e3ebffa5d0e13af009473764dd7ea3087c46cf4d0010" gracePeriod=25
Oct 31 03:44:30.445928 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:30.445898    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6"
Oct 31 03:44:30.572655 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:30.571332    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c33afaeb-7d50-4812-99c7-daa2ad29ef13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33afaeb-7d50-4812-99c7-daa2ad29ef13-plugins-config-volume\") pod \"c33afaeb-7d50-4812-99c7-daa2ad29ef13\" (UID: \"c33afaeb-7d50-4812-99c7-daa2ad29ef13\") "
Oct 31 03:44:30.572655 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:30.571562    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c33afaeb-7d50-4812-99c7-daa2ad29ef13-kube-api-access-gx8cx\" (UniqueName: \"kubernetes.io/projected/c33afaeb-7d50-4812-99c7-daa2ad29ef13-kube-api-access-gx8cx\") pod \"c33afaeb-7d50-4812-99c7-daa2ad29ef13\" (UID: \"c33afaeb-7d50-4812-99c7-daa2ad29ef13\") "
Oct 31 03:44:30.573317 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:30.573287    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c33afaeb-7d50-4812-99c7-daa2ad29ef13-plugins-config-volume" pod "c33afaeb-7d50-4812-99c7-daa2ad29ef13" (UID: "c33afaeb-7d50-4812-99c7-daa2ad29ef13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:44:30.577056 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:30.577033    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33afaeb-7d50-4812-99c7-daa2ad29ef13-kube-api-access-gx8cx" pod "c33afaeb-7d50-4812-99c7-daa2ad29ef13" (UID: "c33afaeb-7d50-4812-99c7-daa2ad29ef13"). InnerVolumeSpecName "kube-api-access-gx8cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:44:30.674076 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:30.673991    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33afaeb-7d50-4812-99c7-daa2ad29ef13-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:44:30.674076 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:30.674022    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx8cx\" (UniqueName: \"kubernetes.io/projected/c33afaeb-7d50-4812-99c7-daa2ad29ef13-kube-api-access-gx8cx\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:44:31.188493 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.188460    4447 generic.go:356] "Generic (PLEG): container finished" podID="c33afaeb-7d50-4812-99c7-daa2ad29ef13" containerID="be9bbabc11bbe3079fd3e3ebffa5d0e13af009473764dd7ea3087c46cf4d0010" exitCode=0
Oct 31 03:44:31.189022 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.188506    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6"
Oct 31 03:44:31.189022 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.188547    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6" event={"ID":"c33afaeb-7d50-4812-99c7-daa2ad29ef13","Type":"ContainerDied","Data":"be9bbabc11bbe3079fd3e3ebffa5d0e13af009473764dd7ea3087c46cf4d0010"}
Oct 31 03:44:31.189022 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.188610    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6" event={"ID":"c33afaeb-7d50-4812-99c7-daa2ad29ef13","Type":"ContainerDied","Data":"d9c180eaa1a48d2f31af2e62ad594c230f38d60d93b6b7ac99fcd8bea942be35"}
Oct 31 03:44:31.189022 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.188640    4447 scope.go:122] "RemoveContainer" containerID="be9bbabc11bbe3079fd3e3ebffa5d0e13af009473764dd7ea3087c46cf4d0010"
Oct 31 03:44:31.199968 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.199952    4447 scope.go:122] "RemoveContainer" containerID="be9bbabc11bbe3079fd3e3ebffa5d0e13af009473764dd7ea3087c46cf4d0010"
Oct 31 03:44:31.200273 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:44:31.200252    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9bbabc11bbe3079fd3e3ebffa5d0e13af009473764dd7ea3087c46cf4d0010\": container with ID starting with be9bbabc11bbe3079fd3e3ebffa5d0e13af009473764dd7ea3087c46cf4d0010 not found: ID does not exist" containerID="be9bbabc11bbe3079fd3e3ebffa5d0e13af009473764dd7ea3087c46cf4d0010"
Oct 31 03:44:31.200336 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.200286    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9bbabc11bbe3079fd3e3ebffa5d0e13af009473764dd7ea3087c46cf4d0010"} err="failed to get container status \"be9bbabc11bbe3079fd3e3ebffa5d0e13af009473764dd7ea3087c46cf4d0010\": rpc error: code = NotFound desc = could not find container \"be9bbabc11bbe3079fd3e3ebffa5d0e13af009473764dd7ea3087c46cf4d0010\": container with ID starting with be9bbabc11bbe3079fd3e3ebffa5d0e13af009473764dd7ea3087c46cf4d0010 not found: ID does not exist"
Oct 31 03:44:31.357481 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.357450    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6"]
Oct 31 03:44:31.362193 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.362164    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e9fd0a85-6bdf8dbb5f-zpjf6"]
Oct 31 03:44:31.755464 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.755433    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb"]
Oct 31 03:44:31.756276 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.756260    4447 cpu_manager.go:414] "containerMap: removing container" podUID="6e1b9359-2dbe-4916-b124-95f4c793db43" containerName="payload-processing"
Oct 31 03:44:31.756351 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.756278    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e1b9359-2dbe-4916-b124-95f4c793db43" containerName="payload-processing"
Oct 31 03:44:31.756351 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.756296    4447 cpu_manager.go:414] "containerMap: removing container" podUID="c33afaeb-7d50-4812-99c7-daa2ad29ef13" containerName="payload-processing"
Oct 31 03:44:31.756351 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.756301    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c33afaeb-7d50-4812-99c7-daa2ad29ef13" containerName="payload-processing"
Oct 31 03:44:31.756351 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.756321    4447 cpu_manager.go:414] "containerMap: removing container" podUID="b1f11045-ca77-415b-893f-88d842338681" containerName="payload-pre-processing"
Oct 31 03:44:31.756351 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.756326    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1f11045-ca77-415b-893f-88d842338681" containerName="payload-pre-processing"
Oct 31 03:44:31.756351 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.756333    4447 cpu_manager.go:414] "containerMap: removing container" podUID="00137945-5937-4cfb-8d79-99dfa03a21e7" containerName="payload-processing"
Oct 31 03:44:31.756351 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.756338    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00137945-5937-4cfb-8d79-99dfa03a21e7" containerName="payload-processing"
Oct 31 03:44:31.756581 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.756404    4447 cpu_manager.go:414] "containerMap: removing container" podUID="d6c0742f-4434-49f0-b2ea-7264667d2933" containerName="payload-processing"
Oct 31 03:44:31.756581 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.756429    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c0742f-4434-49f0-b2ea-7264667d2933" containerName="payload-processing"
Oct 31 03:44:31.756581 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.756451    4447 cpu_manager.go:414] "containerMap: removing container" podUID="668b2bb1-f5a0-454a-bcb5-c28d56aeab40" containerName="payload-pre-processing"
Oct 31 03:44:31.756581 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.756457    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="668b2bb1-f5a0-454a-bcb5-c28d56aeab40" containerName="payload-pre-processing"
Oct 31 03:44:31.757052 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.757027    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1f11045-ca77-415b-893f-88d842338681" containerName="payload-pre-processing"
Oct 31 03:44:31.757532 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.757513    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="00137945-5937-4cfb-8d79-99dfa03a21e7" containerName="payload-processing"
Oct 31 03:44:31.757704 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.757686    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c0742f-4434-49f0-b2ea-7264667d2933" containerName="payload-processing"
Oct 31 03:44:31.757704 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.757705    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="668b2bb1-f5a0-454a-bcb5-c28d56aeab40" containerName="payload-pre-processing"
Oct 31 03:44:31.757852 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.757723    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e1b9359-2dbe-4916-b124-95f4c793db43" containerName="payload-processing"
Oct 31 03:44:31.757852 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.757735    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="c33afaeb-7d50-4812-99c7-daa2ad29ef13" containerName="payload-processing"
Oct 31 03:44:31.761789 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.761766    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb"
Oct 31 03:44:31.816823 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.816791    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-bb362142\""
Oct 31 03:44:31.817112 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.816964    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-bb362142-dockercfg-4jkz8\""
Oct 31 03:44:31.819241 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.819214    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb"]
Oct 31 03:44:31.897448 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.897411    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6bfm\" (UniqueName: \"kubernetes.io/projected/b37925d4-a955-4e16-a0bc-2e9e98ae3d60-kube-api-access-m6bfm\") pod \"payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb\" (UID: \"b37925d4-a955-4e16-a0bc-2e9e98ae3d60\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb"
Oct 31 03:44:31.897707 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:31.897690    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37925d4-a955-4e16-a0bc-2e9e98ae3d60-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb\" (UID: \"b37925d4-a955-4e16-a0bc-2e9e98ae3d60\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb"
Oct 31 03:44:32.004270 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:32.003079    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37925d4-a955-4e16-a0bc-2e9e98ae3d60-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb\" (UID: \"b37925d4-a955-4e16-a0bc-2e9e98ae3d60\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb"
Oct 31 03:44:32.004749 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:32.004726    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6bfm\" (UniqueName: \"kubernetes.io/projected/b37925d4-a955-4e16-a0bc-2e9e98ae3d60-kube-api-access-m6bfm\") pod \"payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb\" (UID: \"b37925d4-a955-4e16-a0bc-2e9e98ae3d60\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb"
Oct 31 03:44:32.005875 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:32.005811    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37925d4-a955-4e16-a0bc-2e9e98ae3d60-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb\" (UID: \"b37925d4-a955-4e16-a0bc-2e9e98ae3d60\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb"
Oct 31 03:44:32.066719 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:32.066690    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6bfm\" (UniqueName: \"kubernetes.io/projected/b37925d4-a955-4e16-a0bc-2e9e98ae3d60-kube-api-access-m6bfm\") pod \"payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb\" (UID: \"b37925d4-a955-4e16-a0bc-2e9e98ae3d60\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb"
Oct 31 03:44:32.079561 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:32.079532    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb"
Oct 31 03:44:32.532215 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:44:32.532187    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37925d4_a955_4e16_a0bc_2e9e98ae3d60.slice/crio-aab2c20d9d5f2722472827b8b8f797b3e1255f6a82866eff2f1c2a679d0c657c WatchSource:0}: Error finding container aab2c20d9d5f2722472827b8b8f797b3e1255f6a82866eff2f1c2a679d0c657c: Status 404 returned error can't find the container with id aab2c20d9d5f2722472827b8b8f797b3e1255f6a82866eff2f1c2a679d0c657c
Oct 31 03:44:32.532539 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:32.532256    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb"]
Oct 31 03:44:32.618553 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:32.618524    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c33afaeb-7d50-4812-99c7-daa2ad29ef13" path="/var/lib/kubelet/pods/c33afaeb-7d50-4812-99c7-daa2ad29ef13/volumes"
Oct 31 03:44:33.215233 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:33.215200    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb" event={"ID":"b37925d4-a955-4e16-a0bc-2e9e98ae3d60","Type":"ContainerStarted","Data":"79e917607a0253514b18e20cd3a1e36a6af192f2d4e4c29a677b3933a7cd21ad"}
Oct 31 03:44:33.215420 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:33.215252    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb" event={"ID":"b37925d4-a955-4e16-a0bc-2e9e98ae3d60","Type":"ContainerStarted","Data":"aab2c20d9d5f2722472827b8b8f797b3e1255f6a82866eff2f1c2a679d0c657c"}
Oct 31 03:44:33.215420 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:33.215274    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb"
Oct 31 03:44:33.333513 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:33.333457    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb" podStartSLOduration=2.333442307 podStartE2EDuration="2.333442307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:33.331727761 +0000 UTC m=+1498.842710163" watchObservedRunningTime="2026-10-31 03:44:33.333442307 +0000 UTC m=+1498.844424684"
Oct 31 03:44:34.847851 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:34.847824    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 03:44:34.864300 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:34.864274    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 03:44:36.686375 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:36.685614    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd"]
Oct 31 03:44:36.694001 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:36.693973    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd"
Oct 31 03:44:36.772099 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:36.770724    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd"]
Oct 31 03:44:36.791377 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:36.791343    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-8de398-dockercfg-5hzf2\""
Oct 31 03:44:36.791554 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:36.791384    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-8de398\""
Oct 31 03:44:36.805312 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:36.805274    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2rd8\" (UniqueName: \"kubernetes.io/projected/994071b8-58c4-4431-8154-c00e67c7b065-kube-api-access-n2rd8\") pod \"payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd\" (UID: \"994071b8-58c4-4431-8154-c00e67c7b065\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd"
Oct 31 03:44:36.805510 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:36.805347    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/994071b8-58c4-4431-8154-c00e67c7b065-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd\" (UID: \"994071b8-58c4-4431-8154-c00e67c7b065\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd"
Oct 31 03:44:36.834955 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:36.834916    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z"]
Oct 31 03:44:36.840508 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:36.840482    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z"
Oct 31 03:44:36.899361 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:36.899278    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z"]
Oct 31 03:44:36.907139 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:36.907109    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a17f12-6a7b-4645-b519-ea369877724d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z\" (UID: \"58a17f12-6a7b-4645-b519-ea369877724d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z"
Oct 31 03:44:36.907492 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:36.907468    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh6jz\" (UniqueName: \"kubernetes.io/projected/58a17f12-6a7b-4645-b519-ea369877724d-kube-api-access-kh6jz\") pod \"payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z\" (UID: \"58a17f12-6a7b-4645-b519-ea369877724d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z"
Oct 31 03:44:36.907811 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:36.907786    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2rd8\" (UniqueName: \"kubernetes.io/projected/994071b8-58c4-4431-8154-c00e67c7b065-kube-api-access-n2rd8\") pod \"payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd\" (UID: \"994071b8-58c4-4431-8154-c00e67c7b065\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd"
Oct 31 03:44:36.908008 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:36.907851    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/994071b8-58c4-4431-8154-c00e67c7b065-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd\" (UID: \"994071b8-58c4-4431-8154-c00e67c7b065\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd"
Oct 31 03:44:36.921909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:36.908851    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/994071b8-58c4-4431-8154-c00e67c7b065-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd\" (UID: \"994071b8-58c4-4431-8154-c00e67c7b065\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd"
Oct 31 03:44:36.980644 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:36.980613    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2rd8\" (UniqueName: \"kubernetes.io/projected/994071b8-58c4-4431-8154-c00e67c7b065-kube-api-access-n2rd8\") pod \"payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd\" (UID: \"994071b8-58c4-4431-8154-c00e67c7b065\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd"
Oct 31 03:44:37.009659 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:37.009613    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a17f12-6a7b-4645-b519-ea369877724d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z\" (UID: \"58a17f12-6a7b-4645-b519-ea369877724d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z"
Oct 31 03:44:37.009866 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:37.009736    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh6jz\" (UniqueName: \"kubernetes.io/projected/58a17f12-6a7b-4645-b519-ea369877724d-kube-api-access-kh6jz\") pod \"payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z\" (UID: \"58a17f12-6a7b-4645-b519-ea369877724d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z"
Oct 31 03:44:37.010451 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:37.010426    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a17f12-6a7b-4645-b519-ea369877724d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z\" (UID: \"58a17f12-6a7b-4645-b519-ea369877724d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z"
Oct 31 03:44:37.024730 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:37.024685    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd"
Oct 31 03:44:37.080908 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:37.076584    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh6jz\" (UniqueName: \"kubernetes.io/projected/58a17f12-6a7b-4645-b519-ea369877724d-kube-api-access-kh6jz\") pod \"payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z\" (UID: \"58a17f12-6a7b-4645-b519-ea369877724d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z"
Oct 31 03:44:37.162471 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:37.162206    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z"
Oct 31 03:44:37.442216 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:44:37.442187    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994071b8_58c4_4431_8154_c00e67c7b065.slice/crio-fecba42a1d5072c65154e9c684f8324e6779a15de29d3edda03cd98cbc07fc05 WatchSource:0}: Error finding container fecba42a1d5072c65154e9c684f8324e6779a15de29d3edda03cd98cbc07fc05: Status 404 returned error can't find the container with id fecba42a1d5072c65154e9c684f8324e6779a15de29d3edda03cd98cbc07fc05
Oct 31 03:44:37.442750 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:37.442632    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd"]
Oct 31 03:44:37.571167 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:44:37.571138    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a17f12_6a7b_4645_b519_ea369877724d.slice/crio-644be4b28c4a695ba3e495a3cc3710128297e6aa23ba97ca9c13d9a76e9eb61a WatchSource:0}: Error finding container 644be4b28c4a695ba3e495a3cc3710128297e6aa23ba97ca9c13d9a76e9eb61a: Status 404 returned error can't find the container with id 644be4b28c4a695ba3e495a3cc3710128297e6aa23ba97ca9c13d9a76e9eb61a
Oct 31 03:44:37.571598 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:37.571578    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z"]
Oct 31 03:44:38.274017 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:38.273969    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z" event={"ID":"58a17f12-6a7b-4645-b519-ea369877724d","Type":"ContainerStarted","Data":"d2dd0312e5dfb336b619de12a80823e2aad780ea74ac5e8fa82cb13d54f0ccbb"}
Oct 31 03:44:38.274563 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:38.274041    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z" event={"ID":"58a17f12-6a7b-4645-b519-ea369877724d","Type":"ContainerStarted","Data":"644be4b28c4a695ba3e495a3cc3710128297e6aa23ba97ca9c13d9a76e9eb61a"}
Oct 31 03:44:38.274563 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:38.274072    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z"
Oct 31 03:44:38.276100 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:38.276073    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd" event={"ID":"994071b8-58c4-4431-8154-c00e67c7b065","Type":"ContainerStarted","Data":"b48f407747e1ac6b6eb828b59e4a29dc9e38e6661a6d2608b11d88d399229c99"}
Oct 31 03:44:38.276230 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:38.276124    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd" event={"ID":"994071b8-58c4-4431-8154-c00e67c7b065","Type":"ContainerStarted","Data":"fecba42a1d5072c65154e9c684f8324e6779a15de29d3edda03cd98cbc07fc05"}
Oct 31 03:44:38.276230 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:38.276154    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd"
Oct 31 03:44:38.395171 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:38.395124    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z" podStartSLOduration=2.395110089 podStartE2EDuration="2.395110089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:38.391039064 +0000 UTC m=+1503.902021441" watchObservedRunningTime="2026-10-31 03:44:38.395110089 +0000 UTC m=+1503.906092465"
Oct 31 03:44:38.509872 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:38.509816    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd" podStartSLOduration=2.509800646 podStartE2EDuration="2.509800646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:38.507623515 +0000 UTC m=+1504.018605892" watchObservedRunningTime="2026-10-31 03:44:38.509800646 +0000 UTC m=+1504.020783047"
Oct 31 03:44:44.226730 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:44.226703    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb"
Oct 31 03:44:49.011043 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:44:49.011005    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:44:49.287409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:49.287331    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z"
Oct 31 03:44:49.287409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:49.287401    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd"
Oct 31 03:44:59.373552 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:59.373511    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"]
Oct 31 03:44:59.373948 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:59.373719    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w" podUID="6be8a119-cf39-497e-b2c4-c13113f46c16" containerName="maas-api" containerID="cri-o://aec89b3a1d59368f6403270a864879c31a60b2ed6b969abf1d330d14b29fdd8a" gracePeriod=30
Oct 31 03:44:59.694900 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:59.694859    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"
Oct 31 03:44:59.773655 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:59.773625    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6be8a119-cf39-497e-b2c4-c13113f46c16-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6be8a119-cf39-497e-b2c4-c13113f46c16-metrics-certs\") pod \"6be8a119-cf39-497e-b2c4-c13113f46c16\" (UID: \"6be8a119-cf39-497e-b2c4-c13113f46c16\") "
Oct 31 03:44:59.773904 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:59.773870    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6be8a119-cf39-497e-b2c4-c13113f46c16-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be8a119-cf39-497e-b2c4-c13113f46c16-maas-api-tls\") pod \"6be8a119-cf39-497e-b2c4-c13113f46c16\" (UID: \"6be8a119-cf39-497e-b2c4-c13113f46c16\") "
Oct 31 03:44:59.774084 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:59.774060    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be8a119-cf39-497e-b2c4-c13113f46c16-kube-api-access-77kd2\" (UniqueName: \"kubernetes.io/projected/6be8a119-cf39-497e-b2c4-c13113f46c16-kube-api-access-77kd2\") pod \"6be8a119-cf39-497e-b2c4-c13113f46c16\" (UID: \"6be8a119-cf39-497e-b2c4-c13113f46c16\") "
Oct 31 03:44:59.774219 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:59.774199    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6be8a119-cf39-497e-b2c4-c13113f46c16-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be8a119-cf39-497e-b2c4-c13113f46c16-openshift-service-ca\") pod \"6be8a119-cf39-497e-b2c4-c13113f46c16\" (UID: \"6be8a119-cf39-497e-b2c4-c13113f46c16\") "
Oct 31 03:44:59.774607 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:59.774567    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be8a119-cf39-497e-b2c4-c13113f46c16-openshift-service-ca" pod "6be8a119-cf39-497e-b2c4-c13113f46c16" (UID: "6be8a119-cf39-497e-b2c4-c13113f46c16"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:44:59.775713 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:59.775684    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be8a119-cf39-497e-b2c4-c13113f46c16-openshift-service-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:44:59.776626 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:59.776599    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be8a119-cf39-497e-b2c4-c13113f46c16-metrics-certs" pod "6be8a119-cf39-497e-b2c4-c13113f46c16" (UID: "6be8a119-cf39-497e-b2c4-c13113f46c16"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:44:59.777405 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:59.777378    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be8a119-cf39-497e-b2c4-c13113f46c16-maas-api-tls" pod "6be8a119-cf39-497e-b2c4-c13113f46c16" (UID: "6be8a119-cf39-497e-b2c4-c13113f46c16"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:44:59.777405 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:59.777391    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be8a119-cf39-497e-b2c4-c13113f46c16-kube-api-access-77kd2" pod "6be8a119-cf39-497e-b2c4-c13113f46c16" (UID: "6be8a119-cf39-497e-b2c4-c13113f46c16"). InnerVolumeSpecName "kube-api-access-77kd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:44:59.877438 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:59.877402    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be8a119-cf39-497e-b2c4-c13113f46c16-maas-api-tls\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:44:59.877438 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:59.877436    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77kd2\" (UniqueName: \"kubernetes.io/projected/6be8a119-cf39-497e-b2c4-c13113f46c16-kube-api-access-77kd2\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:44:59.877638 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:44:59.877448    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6be8a119-cf39-497e-b2c4-c13113f46c16-metrics-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:45:00.260312 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.260277    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz"]
Oct 31 03:45:00.261138 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.261116    4447 cpu_manager.go:414] "containerMap: removing container" podUID="6be8a119-cf39-497e-b2c4-c13113f46c16" containerName="maas-api"
Oct 31 03:45:00.261138 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.261133    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be8a119-cf39-497e-b2c4-c13113f46c16" containerName="maas-api"
Oct 31 03:45:00.261294 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.261274    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be8a119-cf39-497e-b2c4-c13113f46c16" containerName="maas-api"
Oct 31 03:45:00.265310 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.265287    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz"
Oct 31 03:45:00.281619 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.281594    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq"]
Oct 31 03:45:00.345909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.329102    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p"]
Oct 31 03:45:00.357963 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.352039    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq"
Oct 31 03:45:00.389542 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.377315    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 03:45:00.389542 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.387923    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzbgp\" (UniqueName: \"kubernetes.io/projected/43d26fd6-e12c-4144-9b06-1144d5beb09c-kube-api-access-hzbgp\") pod \"maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz\" (UID: \"43d26fd6-e12c-4144-9b06-1144d5beb09c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz"
Oct 31 03:45:00.389542 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.388035    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d26fd6-e12c-4144-9b06-1144d5beb09c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz\" (UID: \"43d26fd6-e12c-4144-9b06-1144d5beb09c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz"
Oct 31 03:45:00.415917 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.395058    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl"]
Oct 31 03:45:00.415917 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.400625    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p"
Oct 31 03:45:00.415917 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.403266    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz"]
Oct 31 03:45:00.427916 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.419079    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl"
Oct 31 03:45:00.427916 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.425339    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq"]
Oct 31 03:45:00.452909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.434680    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p"]
Oct 31 03:45:00.501710 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.494145    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzbgp\" (UniqueName: \"kubernetes.io/projected/43d26fd6-e12c-4144-9b06-1144d5beb09c-kube-api-access-hzbgp\") pod \"maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz\" (UID: \"43d26fd6-e12c-4144-9b06-1144d5beb09c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz"
Oct 31 03:45:00.501710 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.499507    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdef4c7-b0f9-4058-8629-700107f59688-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl\" (UID: \"3fdef4c7-b0f9-4058-8629-700107f59688\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl"
Oct 31 03:45:00.501710 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.499568    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f54eea24-bcc0-4c11-a687-351240b1adb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890305-2jkgq\" (UID: \"f54eea24-bcc0-4c11-a687-351240b1adb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq"
Oct 31 03:45:00.501710 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.499652    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d26fd6-e12c-4144-9b06-1144d5beb09c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz\" (UID: \"43d26fd6-e12c-4144-9b06-1144d5beb09c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz"
Oct 31 03:45:00.501710 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.499687    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d035a15-aefe-450f-8f99-bba3c3f2c15c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p\" (UID: \"4d035a15-aefe-450f-8f99-bba3c3f2c15c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p"
Oct 31 03:45:00.501710 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.499755    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zsgv\" (UniqueName: \"kubernetes.io/projected/4d035a15-aefe-450f-8f99-bba3c3f2c15c-kube-api-access-5zsgv\") pod \"maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p\" (UID: \"4d035a15-aefe-450f-8f99-bba3c3f2c15c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p"
Oct 31 03:45:00.501710 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.499787    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zjwv\" (UniqueName: \"kubernetes.io/projected/3fdef4c7-b0f9-4058-8629-700107f59688-kube-api-access-6zjwv\") pod \"maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl\" (UID: \"3fdef4c7-b0f9-4058-8629-700107f59688\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl"
Oct 31 03:45:00.501710 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.499846    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhcvg\" (UniqueName: \"kubernetes.io/projected/f54eea24-bcc0-4c11-a687-351240b1adb8-kube-api-access-qhcvg\") pod \"maas-api-key-cleanup-29890305-2jkgq\" (UID: \"f54eea24-bcc0-4c11-a687-351240b1adb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq"
Oct 31 03:45:00.501710 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.500898    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d26fd6-e12c-4144-9b06-1144d5beb09c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz\" (UID: \"43d26fd6-e12c-4144-9b06-1144d5beb09c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz"
Oct 31 03:45:00.501710 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.501164    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl"]
Oct 31 03:45:00.509928 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.508612    4447 generic.go:356] "Generic (PLEG): container finished" podID="6be8a119-cf39-497e-b2c4-c13113f46c16" containerID="aec89b3a1d59368f6403270a864879c31a60b2ed6b969abf1d330d14b29fdd8a" exitCode=0
Oct 31 03:45:00.509928 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.508678    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"
Oct 31 03:45:00.509928 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.508943    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w" event={"ID":"6be8a119-cf39-497e-b2c4-c13113f46c16","Type":"ContainerDied","Data":"aec89b3a1d59368f6403270a864879c31a60b2ed6b969abf1d330d14b29fdd8a"}
Oct 31 03:45:00.509928 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.508997    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w" event={"ID":"6be8a119-cf39-497e-b2c4-c13113f46c16","Type":"ContainerDied","Data":"63278f6651c608f9f95cb153e9fc40699bf869061a76eea8e0b9a78de4e7e984"}
Oct 31 03:45:00.509928 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.509037    4447 scope.go:122] "RemoveContainer" containerID="aec89b3a1d59368f6403270a864879c31a60b2ed6b969abf1d330d14b29fdd8a"
Oct 31 03:45:00.538525 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.538493    4447 scope.go:122] "RemoveContainer" containerID="aec89b3a1d59368f6403270a864879c31a60b2ed6b969abf1d330d14b29fdd8a"
Oct 31 03:45:00.539143 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:45:00.539113    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aec89b3a1d59368f6403270a864879c31a60b2ed6b969abf1d330d14b29fdd8a\": container with ID starting with aec89b3a1d59368f6403270a864879c31a60b2ed6b969abf1d330d14b29fdd8a not found: ID does not exist" containerID="aec89b3a1d59368f6403270a864879c31a60b2ed6b969abf1d330d14b29fdd8a"
Oct 31 03:45:00.539312 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.539153    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec89b3a1d59368f6403270a864879c31a60b2ed6b969abf1d330d14b29fdd8a"} err="failed to get container status \"aec89b3a1d59368f6403270a864879c31a60b2ed6b969abf1d330d14b29fdd8a\": rpc error: code = NotFound desc = could not find container \"aec89b3a1d59368f6403270a864879c31a60b2ed6b969abf1d330d14b29fdd8a\": container with ID starting with aec89b3a1d59368f6403270a864879c31a60b2ed6b969abf1d330d14b29fdd8a not found: ID does not exist"
Oct 31 03:45:00.560466 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.560439    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzbgp\" (UniqueName: \"kubernetes.io/projected/43d26fd6-e12c-4144-9b06-1144d5beb09c-kube-api-access-hzbgp\") pod \"maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz\" (UID: \"43d26fd6-e12c-4144-9b06-1144d5beb09c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz"
Oct 31 03:45:00.585144 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.584151    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz"
Oct 31 03:45:00.601802 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.601774    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zsgv\" (UniqueName: \"kubernetes.io/projected/4d035a15-aefe-450f-8f99-bba3c3f2c15c-kube-api-access-5zsgv\") pod \"maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p\" (UID: \"4d035a15-aefe-450f-8f99-bba3c3f2c15c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p"
Oct 31 03:45:00.601983 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.601835    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zjwv\" (UniqueName: \"kubernetes.io/projected/3fdef4c7-b0f9-4058-8629-700107f59688-kube-api-access-6zjwv\") pod \"maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl\" (UID: \"3fdef4c7-b0f9-4058-8629-700107f59688\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl"
Oct 31 03:45:00.601983 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.601930    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhcvg\" (UniqueName: \"kubernetes.io/projected/f54eea24-bcc0-4c11-a687-351240b1adb8-kube-api-access-qhcvg\") pod \"maas-api-key-cleanup-29890305-2jkgq\" (UID: \"f54eea24-bcc0-4c11-a687-351240b1adb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq"
Oct 31 03:45:00.602154 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.602140    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdef4c7-b0f9-4058-8629-700107f59688-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl\" (UID: \"3fdef4c7-b0f9-4058-8629-700107f59688\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl"
Oct 31 03:45:00.602214 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.602187    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f54eea24-bcc0-4c11-a687-351240b1adb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890305-2jkgq\" (UID: \"f54eea24-bcc0-4c11-a687-351240b1adb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq"
Oct 31 03:45:00.602296 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.602284    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d035a15-aefe-450f-8f99-bba3c3f2c15c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p\" (UID: \"4d035a15-aefe-450f-8f99-bba3c3f2c15c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p"
Oct 31 03:45:00.605507 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.604399    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdef4c7-b0f9-4058-8629-700107f59688-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl\" (UID: \"3fdef4c7-b0f9-4058-8629-700107f59688\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl"
Oct 31 03:45:00.605507 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.604614    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d035a15-aefe-450f-8f99-bba3c3f2c15c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p\" (UID: \"4d035a15-aefe-450f-8f99-bba3c3f2c15c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p"
Oct 31 03:45:00.605507 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.605427    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f54eea24-bcc0-4c11-a687-351240b1adb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890305-2jkgq\" (UID: \"f54eea24-bcc0-4c11-a687-351240b1adb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq"
Oct 31 03:45:00.687688 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.687650    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zsgv\" (UniqueName: \"kubernetes.io/projected/4d035a15-aefe-450f-8f99-bba3c3f2c15c-kube-api-access-5zsgv\") pod \"maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p\" (UID: \"4d035a15-aefe-450f-8f99-bba3c3f2c15c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p"
Oct 31 03:45:00.690530 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.690502    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zjwv\" (UniqueName: \"kubernetes.io/projected/3fdef4c7-b0f9-4058-8629-700107f59688-kube-api-access-6zjwv\") pod \"maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl\" (UID: \"3fdef4c7-b0f9-4058-8629-700107f59688\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl"
Oct 31 03:45:00.702773 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.702748    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhcvg\" (UniqueName: \"kubernetes.io/projected/f54eea24-bcc0-4c11-a687-351240b1adb8-kube-api-access-qhcvg\") pod \"maas-api-key-cleanup-29890305-2jkgq\" (UID: \"f54eea24-bcc0-4c11-a687-351240b1adb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq"
Oct 31 03:45:00.826372 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.826294    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq"
Oct 31 03:45:00.870453 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.870425    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p"
Oct 31 03:45:00.900286 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.900262    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl"
Oct 31 03:45:00.981534 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.981460    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"]
Oct 31 03:45:00.990024 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.989588    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz"]
Oct 31 03:45:00.995234 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:00.995206    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e9fd0a85-6d9d8bdddf-dd74w"]
Oct 31 03:45:01.215095 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:45:01.215066    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54eea24_bcc0_4c11_a687_351240b1adb8.slice/crio-cdc80d6f7601fe2a45ad6f1b00e92a1c418315f88abf6a2d9c6e44804a44d534 WatchSource:0}: Error finding container cdc80d6f7601fe2a45ad6f1b00e92a1c418315f88abf6a2d9c6e44804a44d534: Status 404 returned error can't find the container with id cdc80d6f7601fe2a45ad6f1b00e92a1c418315f88abf6a2d9c6e44804a44d534
Oct 31 03:45:01.215539 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:01.215512    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq"]
Oct 31 03:45:01.266296 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:45:01.266127    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d035a15_aefe_450f_8f99_bba3c3f2c15c.slice/crio-fd55867ed4ab5cbec4903e8424e90a866d9bd597ea690ddaba600690bdd191d0 WatchSource:0}: Error finding container fd55867ed4ab5cbec4903e8424e90a866d9bd597ea690ddaba600690bdd191d0: Status 404 returned error can't find the container with id fd55867ed4ab5cbec4903e8424e90a866d9bd597ea690ddaba600690bdd191d0
Oct 31 03:45:01.267555 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:01.267528    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p"]
Oct 31 03:45:01.309985 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:01.309958    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl"]
Oct 31 03:45:01.378089 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:45:01.378064    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fdef4c7_b0f9_4058_8629_700107f59688.slice/crio-4dbc7aeca75adede43aa46afaa7f75285f98e65d2b7f15ec1b439411206405c8 WatchSource:0}: Error finding container 4dbc7aeca75adede43aa46afaa7f75285f98e65d2b7f15ec1b439411206405c8: Status 404 returned error can't find the container with id 4dbc7aeca75adede43aa46afaa7f75285f98e65d2b7f15ec1b439411206405c8
Oct 31 03:45:01.522094 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:01.522058    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p" event={"ID":"4d035a15-aefe-450f-8f99-bba3c3f2c15c","Type":"ContainerStarted","Data":"fd55867ed4ab5cbec4903e8424e90a866d9bd597ea690ddaba600690bdd191d0"}
Oct 31 03:45:01.523966 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:01.523926    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl" event={"ID":"3fdef4c7-b0f9-4058-8629-700107f59688","Type":"ContainerStarted","Data":"4dbc7aeca75adede43aa46afaa7f75285f98e65d2b7f15ec1b439411206405c8"}
Oct 31 03:45:01.526617 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:01.526594    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq" event={"ID":"f54eea24-bcc0-4c11-a687-351240b1adb8","Type":"ContainerStarted","Data":"cdc80d6f7601fe2a45ad6f1b00e92a1c418315f88abf6a2d9c6e44804a44d534"}
Oct 31 03:45:01.530662 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:01.530634    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz" event={"ID":"43d26fd6-e12c-4144-9b06-1144d5beb09c","Type":"ContainerStarted","Data":"a6c5b5a6a1fcb07391697ff2262759fd601a2d9409cec1f1ed93297d3aafc194"}
Oct 31 03:45:01.530860 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:01.530842    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz" event={"ID":"43d26fd6-e12c-4144-9b06-1144d5beb09c","Type":"ContainerStarted","Data":"9af7368e2e9277aad9b5cee9a0334f40652938ae3021cca5f37ed7a13328a531"}
Oct 31 03:45:01.651619 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:01.651555    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz" podStartSLOduration=1.6515363299999999 podStartE2EDuration="1.65153633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:01.646216175 +0000 UTC m=+1527.157198623" watchObservedRunningTime="2026-10-31 03:45:01.65153633 +0000 UTC m=+1527.162518707"
Oct 31 03:45:02.541966 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:02.541932    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p" event={"ID":"4d035a15-aefe-450f-8f99-bba3c3f2c15c","Type":"ContainerStarted","Data":"96b5b76194f4d8842d3e4b8642041b2bec4aa2315afe5ce5449fa7a51c70291c"}
Oct 31 03:45:02.543874 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:02.543839    4447 generic.go:356] "Generic (PLEG): container finished" podID="3fdef4c7-b0f9-4058-8629-700107f59688" containerID="7c6ba67b421d7096cd80512106612d3da03b33a90e1af8305696fede7b70b552" exitCode=60
Oct 31 03:45:02.544022 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:02.543958    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl" event={"ID":"3fdef4c7-b0f9-4058-8629-700107f59688","Type":"ContainerDied","Data":"7c6ba67b421d7096cd80512106612d3da03b33a90e1af8305696fede7b70b552"}
Oct 31 03:45:02.544319 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:02.544285    4447 scope.go:122] "RemoveContainer" containerID="7c6ba67b421d7096cd80512106612d3da03b33a90e1af8305696fede7b70b552"
Oct 31 03:45:02.545996 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:02.545975    4447 generic.go:356] "Generic (PLEG): container finished" podID="f54eea24-bcc0-4c11-a687-351240b1adb8" containerID="7d8e46d132fa6609840bbad9e98f6317b95ae5898f49235b08279d31a7d11829" exitCode=60
Oct 31 03:45:02.546129 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:02.546079    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq" event={"ID":"f54eea24-bcc0-4c11-a687-351240b1adb8","Type":"ContainerDied","Data":"7d8e46d132fa6609840bbad9e98f6317b95ae5898f49235b08279d31a7d11829"}
Oct 31 03:45:02.546342 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:02.546315    4447 scope.go:122] "RemoveContainer" containerID="7d8e46d132fa6609840bbad9e98f6317b95ae5898f49235b08279d31a7d11829"
Oct 31 03:45:02.548417 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:02.548383    4447 generic.go:356] "Generic (PLEG): container finished" podID="43d26fd6-e12c-4144-9b06-1144d5beb09c" containerID="a6c5b5a6a1fcb07391697ff2262759fd601a2d9409cec1f1ed93297d3aafc194" exitCode=60
Oct 31 03:45:02.548518 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:02.548465    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz" event={"ID":"43d26fd6-e12c-4144-9b06-1144d5beb09c","Type":"ContainerDied","Data":"a6c5b5a6a1fcb07391697ff2262759fd601a2d9409cec1f1ed93297d3aafc194"}
Oct 31 03:45:02.548751 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:02.548698    4447 scope.go:122] "RemoveContainer" containerID="a6c5b5a6a1fcb07391697ff2262759fd601a2d9409cec1f1ed93297d3aafc194"
Oct 31 03:45:02.650396 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:02.650365    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be8a119-cf39-497e-b2c4-c13113f46c16" path="/var/lib/kubelet/pods/6be8a119-cf39-497e-b2c4-c13113f46c16/volumes"
Oct 31 03:45:02.662309 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:02.662251    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p" podStartSLOduration=2.6622379130000002 podStartE2EDuration="2.662237913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:02.658125264 +0000 UTC m=+1528.169107649" watchObservedRunningTime="2026-10-31 03:45:02.662237913 +0000 UTC m=+1528.173220314"
Oct 31 03:45:03.558949 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:03.558916    4447 generic.go:356] "Generic (PLEG): container finished" podID="4d035a15-aefe-450f-8f99-bba3c3f2c15c" containerID="96b5b76194f4d8842d3e4b8642041b2bec4aa2315afe5ce5449fa7a51c70291c" exitCode=60
Oct 31 03:45:03.559416 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:03.558980    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p" event={"ID":"4d035a15-aefe-450f-8f99-bba3c3f2c15c","Type":"ContainerDied","Data":"96b5b76194f4d8842d3e4b8642041b2bec4aa2315afe5ce5449fa7a51c70291c"}
Oct 31 03:45:03.559416 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:03.559375    4447 scope.go:122] "RemoveContainer" containerID="96b5b76194f4d8842d3e4b8642041b2bec4aa2315afe5ce5449fa7a51c70291c"
Oct 31 03:45:03.561443 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:03.561405    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl" event={"ID":"3fdef4c7-b0f9-4058-8629-700107f59688","Type":"ContainerStarted","Data":"f60fe7cadcc9ab4ea1ae409f625c8821b8b4d6beaf6fd47154ddfc8b893a1d43"}
Oct 31 03:45:03.563780 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:03.563758    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq" event={"ID":"f54eea24-bcc0-4c11-a687-351240b1adb8","Type":"ContainerStarted","Data":"0eb567a37f00a88597fe7236dace5e812425adb10bd4fcc952fe58705241eb22"}
Oct 31 03:45:03.565771 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:03.565747    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz" event={"ID":"43d26fd6-e12c-4144-9b06-1144d5beb09c","Type":"ContainerStarted","Data":"7e5b2bf983670522595af635935d5bef15628c0782289b5d5581784f4723622f"}
Oct 31 03:45:03.906020 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:03.905971    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl" podStartSLOduration=3.9059559520000002 podStartE2EDuration="3.905955952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:03.904298027 +0000 UTC m=+1529.415280405" watchObservedRunningTime="2026-10-31 03:45:03.905955952 +0000 UTC m=+1529.416938328"
Oct 31 03:45:04.024174 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:04.024123    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq" podStartSLOduration=4.024109797 podStartE2EDuration="4.024109797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:04.021074122 +0000 UTC m=+1529.532056501" watchObservedRunningTime="2026-10-31 03:45:04.024109797 +0000 UTC m=+1529.535092174"
Oct 31 03:45:04.576027 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:04.575994    4447 generic.go:356] "Generic (PLEG): container finished" podID="43d26fd6-e12c-4144-9b06-1144d5beb09c" containerID="7e5b2bf983670522595af635935d5bef15628c0782289b5d5581784f4723622f" exitCode=60
Oct 31 03:45:04.576483 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:04.576129    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz" event={"ID":"43d26fd6-e12c-4144-9b06-1144d5beb09c","Type":"ContainerDied","Data":"7e5b2bf983670522595af635935d5bef15628c0782289b5d5581784f4723622f"}
Oct 31 03:45:04.576483 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:04.576183    4447 scope.go:122] "RemoveContainer" containerID="a6c5b5a6a1fcb07391697ff2262759fd601a2d9409cec1f1ed93297d3aafc194"
Oct 31 03:45:04.576592 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:04.576567    4447 scope.go:122] "RemoveContainer" containerID="7e5b2bf983670522595af635935d5bef15628c0782289b5d5581784f4723622f"
Oct 31 03:45:04.577046 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:45:04.577019    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz_odh-ai-gateway-infra(43d26fd6-e12c-4144-9b06-1144d5beb09c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz" podUID="43d26fd6-e12c-4144-9b06-1144d5beb09c"
Oct 31 03:45:04.578637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:04.578492    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p" event={"ID":"4d035a15-aefe-450f-8f99-bba3c3f2c15c","Type":"ContainerStarted","Data":"7ba925f393dff527c450b5ada06ffa638b88852d58d49f016033c0903533f83a"}
Oct 31 03:45:04.580549 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:04.580527    4447 generic.go:356] "Generic (PLEG): container finished" podID="3fdef4c7-b0f9-4058-8629-700107f59688" containerID="f60fe7cadcc9ab4ea1ae409f625c8821b8b4d6beaf6fd47154ddfc8b893a1d43" exitCode=60
Oct 31 03:45:04.580673 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:04.580606    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl" event={"ID":"3fdef4c7-b0f9-4058-8629-700107f59688","Type":"ContainerDied","Data":"f60fe7cadcc9ab4ea1ae409f625c8821b8b4d6beaf6fd47154ddfc8b893a1d43"}
Oct 31 03:45:04.580852 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:04.580832    4447 scope.go:122] "RemoveContainer" containerID="f60fe7cadcc9ab4ea1ae409f625c8821b8b4d6beaf6fd47154ddfc8b893a1d43"
Oct 31 03:45:04.581162 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:45:04.581141    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl_odh-ai-gateway-infra(3fdef4c7-b0f9-4058-8629-700107f59688)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl" podUID="3fdef4c7-b0f9-4058-8629-700107f59688"
Oct 31 03:45:04.582746 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:04.582723    4447 generic.go:356] "Generic (PLEG): container finished" podID="f54eea24-bcc0-4c11-a687-351240b1adb8" containerID="0eb567a37f00a88597fe7236dace5e812425adb10bd4fcc952fe58705241eb22" exitCode=60
Oct 31 03:45:04.582834 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:04.582747    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq" event={"ID":"f54eea24-bcc0-4c11-a687-351240b1adb8","Type":"ContainerDied","Data":"0eb567a37f00a88597fe7236dace5e812425adb10bd4fcc952fe58705241eb22"}
Oct 31 03:45:04.583116 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:04.583100    4447 scope.go:122] "RemoveContainer" containerID="0eb567a37f00a88597fe7236dace5e812425adb10bd4fcc952fe58705241eb22"
Oct 31 03:45:04.583431 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:45:04.583411    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890305-2jkgq_odh-ai-gateway-infra(f54eea24-bcc0-4c11-a687-351240b1adb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq" podUID="f54eea24-bcc0-4c11-a687-351240b1adb8"
Oct 31 03:45:04.591758 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:04.591742    4447 scope.go:122] "RemoveContainer" containerID="7c6ba67b421d7096cd80512106612d3da03b33a90e1af8305696fede7b70b552"
Oct 31 03:45:04.606597 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:04.606574    4447 scope.go:122] "RemoveContainer" containerID="7d8e46d132fa6609840bbad9e98f6317b95ae5898f49235b08279d31a7d11829"
Oct 31 03:45:05.604567 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:05.604536    4447 scope.go:122] "RemoveContainer" containerID="0eb567a37f00a88597fe7236dace5e812425adb10bd4fcc952fe58705241eb22"
Oct 31 03:45:05.605090 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:45:05.604870    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890305-2jkgq_odh-ai-gateway-infra(f54eea24-bcc0-4c11-a687-351240b1adb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq" podUID="f54eea24-bcc0-4c11-a687-351240b1adb8"
Oct 31 03:45:05.606578 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:05.606551    4447 scope.go:122] "RemoveContainer" containerID="7e5b2bf983670522595af635935d5bef15628c0782289b5d5581784f4723622f"
Oct 31 03:45:05.606940 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:45:05.606853    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz_odh-ai-gateway-infra(43d26fd6-e12c-4144-9b06-1144d5beb09c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz" podUID="43d26fd6-e12c-4144-9b06-1144d5beb09c"
Oct 31 03:45:05.608395 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:05.608370    4447 generic.go:356] "Generic (PLEG): container finished" podID="4d035a15-aefe-450f-8f99-bba3c3f2c15c" containerID="7ba925f393dff527c450b5ada06ffa638b88852d58d49f016033c0903533f83a" exitCode=60
Oct 31 03:45:05.608521 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:05.608453    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p" event={"ID":"4d035a15-aefe-450f-8f99-bba3c3f2c15c","Type":"ContainerDied","Data":"7ba925f393dff527c450b5ada06ffa638b88852d58d49f016033c0903533f83a"}
Oct 31 03:45:05.608590 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:05.608540    4447 scope.go:122] "RemoveContainer" containerID="96b5b76194f4d8842d3e4b8642041b2bec4aa2315afe5ce5449fa7a51c70291c"
Oct 31 03:45:05.610583 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:05.610562    4447 scope.go:122] "RemoveContainer" containerID="7ba925f393dff527c450b5ada06ffa638b88852d58d49f016033c0903533f83a"
Oct 31 03:45:05.610940 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:05.610879    4447 scope.go:122] "RemoveContainer" containerID="f60fe7cadcc9ab4ea1ae409f625c8821b8b4d6beaf6fd47154ddfc8b893a1d43"
Oct 31 03:45:05.611043 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:45:05.610957    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p_odh-ai-gateway-infra(4d035a15-aefe-450f-8f99-bba3c3f2c15c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p" podUID="4d035a15-aefe-450f-8f99-bba3c3f2c15c"
Oct 31 03:45:05.611181 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:45:05.611163    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl_odh-ai-gateway-infra(3fdef4c7-b0f9-4058-8629-700107f59688)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl" podUID="3fdef4c7-b0f9-4058-8629-700107f59688"
Oct 31 03:45:06.621499 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:06.621471    4447 scope.go:122] "RemoveContainer" containerID="7ba925f393dff527c450b5ada06ffa638b88852d58d49f016033c0903533f83a"
Oct 31 03:45:06.621962 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:45:06.621939    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p_odh-ai-gateway-infra(4d035a15-aefe-450f-8f99-bba3c3f2c15c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p" podUID="4d035a15-aefe-450f-8f99-bba3c3f2c15c"
Oct 31 03:45:14.631333 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:14.631249    4447 scope.go:122] "RemoveContainer" containerID="f60fe7cadcc9ab4ea1ae409f625c8821b8b4d6beaf6fd47154ddfc8b893a1d43"
Oct 31 03:45:15.166191 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.166162    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"]
Oct 31 03:45:15.172861 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.172841    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.234934 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.234898    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"]
Oct 31 03:45:15.254162 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.254078    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgnb2\" (UniqueName: \"kubernetes.io/projected/84553d7f-51d4-4b72-b2dd-ee522e5413ad-kube-api-access-lgnb2\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.254327 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.254214    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84553d7f-51d4-4b72-b2dd-ee522e5413ad-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.254327 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.254305    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84553d7f-51d4-4b72-b2dd-ee522e5413ad-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.254451 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.254341    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84553d7f-51d4-4b72-b2dd-ee522e5413ad-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.254451 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.254380    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84553d7f-51d4-4b72-b2dd-ee522e5413ad-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.254557 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.254503    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84553d7f-51d4-4b72-b2dd-ee522e5413ad-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.254630 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.254566    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84553d7f-51d4-4b72-b2dd-ee522e5413ad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.254630 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.254609    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84553d7f-51d4-4b72-b2dd-ee522e5413ad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.254740 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.254629    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84553d7f-51d4-4b72-b2dd-ee522e5413ad-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.356311 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.356276    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgnb2\" (UniqueName: \"kubernetes.io/projected/84553d7f-51d4-4b72-b2dd-ee522e5413ad-kube-api-access-lgnb2\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.356510 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.356371    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84553d7f-51d4-4b72-b2dd-ee522e5413ad-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.356510 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.356431    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84553d7f-51d4-4b72-b2dd-ee522e5413ad-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.356510 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.356463    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84553d7f-51d4-4b72-b2dd-ee522e5413ad-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.356510 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.356509    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84553d7f-51d4-4b72-b2dd-ee522e5413ad-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.356722 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.356560    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84553d7f-51d4-4b72-b2dd-ee522e5413ad-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.356722 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.356605    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84553d7f-51d4-4b72-b2dd-ee522e5413ad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.356722 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.356646    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84553d7f-51d4-4b72-b2dd-ee522e5413ad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.356722 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.356667    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84553d7f-51d4-4b72-b2dd-ee522e5413ad-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.356966 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.356767    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84553d7f-51d4-4b72-b2dd-ee522e5413ad-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.356966 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.356857    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84553d7f-51d4-4b72-b2dd-ee522e5413ad-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.356966 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.356875    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84553d7f-51d4-4b72-b2dd-ee522e5413ad-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.357083 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.356976    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84553d7f-51d4-4b72-b2dd-ee522e5413ad-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.357348 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.357330    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84553d7f-51d4-4b72-b2dd-ee522e5413ad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.359336 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.359314    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84553d7f-51d4-4b72-b2dd-ee522e5413ad-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.359528 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.359512    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84553d7f-51d4-4b72-b2dd-ee522e5413ad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.418040 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.418000    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84553d7f-51d4-4b72-b2dd-ee522e5413ad-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.418214 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.418130    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgnb2\" (UniqueName: \"kubernetes.io/projected/84553d7f-51d4-4b72-b2dd-ee522e5413ad-kube-api-access-lgnb2\") pod \"maas-default-gateway-openshift-default-fd4b99879-dqwsm\" (UID: \"84553d7f-51d4-4b72-b2dd-ee522e5413ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.488641 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.488604    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:15.601812 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.601368    4447 scope.go:122] "RemoveContainer" containerID="0eb567a37f00a88597fe7236dace5e812425adb10bd4fcc952fe58705241eb22"
Oct 31 03:45:15.698284 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.698250    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl" event={"ID":"3fdef4c7-b0f9-4058-8629-700107f59688","Type":"ContainerStarted","Data":"7cc9f39701acbde46cf01ef6a8a52cfcdfc516ba90846d7753257d62807ebeae"}
Oct 31 03:45:15.926968 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:45:15.926935    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84553d7f_51d4_4b72_b2dd_ee522e5413ad.slice/crio-b1e0563e0ac8c6c291efb587e5c1059df0090647018a525566259b81c71aa08a WatchSource:0}: Error finding container b1e0563e0ac8c6c291efb587e5c1059df0090647018a525566259b81c71aa08a: Status 404 returned error can't find the container with id b1e0563e0ac8c6c291efb587e5c1059df0090647018a525566259b81c71aa08a
Oct 31 03:45:15.927995 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.927973    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"]
Oct 31 03:45:15.930449 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.930416    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:45:15.930557 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.930478    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:45:15.930557 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:15.930512    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:45:16.034984 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.034907    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn"]
Oct 31 03:45:16.039828 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.039807    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn"
Oct 31 03:45:16.106623 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.097621    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-bb362142-serving-cert\""
Oct 31 03:45:16.106623 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.102682    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn"]
Oct 31 03:45:16.170119 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.170081    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-metrics-certs\") pod \"maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn\" (UID: \"b0cd404b-1899-4e68-a8d1-09d37d8cb5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn"
Oct 31 03:45:16.170307 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.170217    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qh5j\" (UniqueName: \"kubernetes.io/projected/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-kube-api-access-2qh5j\") pod \"maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn\" (UID: \"b0cd404b-1899-4e68-a8d1-09d37d8cb5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn"
Oct 31 03:45:16.170490 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.170466    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-maas-api-tls\") pod \"maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn\" (UID: \"b0cd404b-1899-4e68-a8d1-09d37d8cb5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn"
Oct 31 03:45:16.170588 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.170571    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-openshift-service-ca\") pod \"maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn\" (UID: \"b0cd404b-1899-4e68-a8d1-09d37d8cb5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn"
Oct 31 03:45:16.272612 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.272573    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-metrics-certs\") pod \"maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn\" (UID: \"b0cd404b-1899-4e68-a8d1-09d37d8cb5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn"
Oct 31 03:45:16.272830 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.272648    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qh5j\" (UniqueName: \"kubernetes.io/projected/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-kube-api-access-2qh5j\") pod \"maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn\" (UID: \"b0cd404b-1899-4e68-a8d1-09d37d8cb5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn"
Oct 31 03:45:16.272830 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.272802    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-maas-api-tls\") pod \"maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn\" (UID: \"b0cd404b-1899-4e68-a8d1-09d37d8cb5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn"
Oct 31 03:45:16.272976 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.272852    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-openshift-service-ca\") pod \"maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn\" (UID: \"b0cd404b-1899-4e68-a8d1-09d37d8cb5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn"
Oct 31 03:45:16.273863 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.273831    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-openshift-service-ca\") pod \"maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn\" (UID: \"b0cd404b-1899-4e68-a8d1-09d37d8cb5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn"
Oct 31 03:45:16.275415 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.275394    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-maas-api-tls\") pod \"maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn\" (UID: \"b0cd404b-1899-4e68-a8d1-09d37d8cb5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn"
Oct 31 03:45:16.275505 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.275455    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-metrics-certs\") pod \"maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn\" (UID: \"b0cd404b-1899-4e68-a8d1-09d37d8cb5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn"
Oct 31 03:45:16.340041 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.339959    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qh5j\" (UniqueName: \"kubernetes.io/projected/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-kube-api-access-2qh5j\") pod \"maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn\" (UID: \"b0cd404b-1899-4e68-a8d1-09d37d8cb5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn"
Oct 31 03:45:16.362170 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.362137    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn"
Oct 31 03:45:16.642325 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.642239    4447 scope.go:122] "RemoveContainer" containerID="7ba925f393dff527c450b5ada06ffa638b88852d58d49f016033c0903533f83a"
Oct 31 03:45:16.642728 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.642698    4447 scope.go:122] "RemoveContainer" containerID="7e5b2bf983670522595af635935d5bef15628c0782289b5d5581784f4723622f"
Oct 31 03:45:16.709381 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.709357    4447 generic.go:356] "Generic (PLEG): container finished" podID="f54eea24-bcc0-4c11-a687-351240b1adb8" containerID="c1df8d846d44d1d9771f5839cc77b69d13adf1facf09e6c4f7d6e1059289b548" exitCode=60
Oct 31 03:45:16.709721 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.709428    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq" event={"ID":"f54eea24-bcc0-4c11-a687-351240b1adb8","Type":"ContainerDied","Data":"c1df8d846d44d1d9771f5839cc77b69d13adf1facf09e6c4f7d6e1059289b548"}
Oct 31 03:45:16.709721 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.709496    4447 scope.go:122] "RemoveContainer" containerID="0eb567a37f00a88597fe7236dace5e812425adb10bd4fcc952fe58705241eb22"
Oct 31 03:45:16.710005 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.709987    4447 scope.go:122] "RemoveContainer" containerID="c1df8d846d44d1d9771f5839cc77b69d13adf1facf09e6c4f7d6e1059289b548"
Oct 31 03:45:16.710415 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:45:16.710392    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890305-2jkgq_odh-ai-gateway-infra(f54eea24-bcc0-4c11-a687-351240b1adb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq" podUID="f54eea24-bcc0-4c11-a687-351240b1adb8"
Oct 31 03:45:16.711468 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.711443    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm" event={"ID":"84553d7f-51d4-4b72-b2dd-ee522e5413ad","Type":"ContainerStarted","Data":"10e0612302964190acc4b45c12820ee8d41dd22d83aad074b90adce6a71cc6a2"}
Oct 31 03:45:16.711573 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.711498    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm" event={"ID":"84553d7f-51d4-4b72-b2dd-ee522e5413ad","Type":"ContainerStarted","Data":"b1e0563e0ac8c6c291efb587e5c1059df0090647018a525566259b81c71aa08a"}
Oct 31 03:45:16.713438 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.713417    4447 generic.go:356] "Generic (PLEG): container finished" podID="3fdef4c7-b0f9-4058-8629-700107f59688" containerID="7cc9f39701acbde46cf01ef6a8a52cfcdfc516ba90846d7753257d62807ebeae" exitCode=60
Oct 31 03:45:16.713539 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.713493    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl" event={"ID":"3fdef4c7-b0f9-4058-8629-700107f59688","Type":"ContainerDied","Data":"7cc9f39701acbde46cf01ef6a8a52cfcdfc516ba90846d7753257d62807ebeae"}
Oct 31 03:45:16.713879 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.713862    4447 scope.go:122] "RemoveContainer" containerID="7cc9f39701acbde46cf01ef6a8a52cfcdfc516ba90846d7753257d62807ebeae"
Oct 31 03:45:16.714260 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:45:16.714233    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl_odh-ai-gateway-infra(3fdef4c7-b0f9-4058-8629-700107f59688)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl" podUID="3fdef4c7-b0f9-4058-8629-700107f59688"
Oct 31 03:45:16.759535 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.759504    4447 scope.go:122] "RemoveContainer" containerID="f60fe7cadcc9ab4ea1ae409f625c8821b8b4d6beaf6fd47154ddfc8b893a1d43"
Oct 31 03:45:16.762363 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.762336    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn"]
Oct 31 03:45:16.826151 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.826114    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl"]
Oct 31 03:45:16.957035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:16.956975    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm" podStartSLOduration=1.956957973 podStartE2EDuration="1.956957973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:16.954224163 +0000 UTC m=+1542.465206541" watchObservedRunningTime="2026-10-31 03:45:16.956957973 +0000 UTC m=+1542.467940350"
Oct 31 03:45:17.491331 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:17.491295    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:17.496326 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:17.496002    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-dqwsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 31 03:45:17.496326 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:17.496064    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm" podUID="84553d7f-51d4-4b72-b2dd-ee522e5413ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 31 03:45:17.727781 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:17.727753    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz" event={"ID":"43d26fd6-e12c-4144-9b06-1144d5beb09c","Type":"ContainerStarted","Data":"462cb52258eebf825549d4895d530883957c49c47e1f25fa4b107c13bbf4eabc"}
Oct 31 03:45:17.729002 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:17.728974    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn" event={"ID":"b0cd404b-1899-4e68-a8d1-09d37d8cb5e7","Type":"ContainerStarted","Data":"809b3af5bf0ddeca02c3742679c99e742ffac39046a63b90ab50dde7ced1d303"}
Oct 31 03:45:17.730999 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:17.730971    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p" event={"ID":"4d035a15-aefe-450f-8f99-bba3c3f2c15c","Type":"ContainerStarted","Data":"ac3db44c5f727d189b19b0fa18520d5479d9bff951aaebd58fce811333e320ff"}
Oct 31 03:45:17.839484 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:17.839453    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq"]
Oct 31 03:45:18.098910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.096959    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl"
Oct 31 03:45:18.124298 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.124273    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq"
Oct 31 03:45:18.225488 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.225450    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f54eea24-bcc0-4c11-a687-351240b1adb8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f54eea24-bcc0-4c11-a687-351240b1adb8-maas-api-service-ca\") pod \"f54eea24-bcc0-4c11-a687-351240b1adb8\" (UID: \"f54eea24-bcc0-4c11-a687-351240b1adb8\") "
Oct 31 03:45:18.225677 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.225610    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f54eea24-bcc0-4c11-a687-351240b1adb8-kube-api-access-qhcvg\" (UniqueName: \"kubernetes.io/projected/f54eea24-bcc0-4c11-a687-351240b1adb8-kube-api-access-qhcvg\") pod \"f54eea24-bcc0-4c11-a687-351240b1adb8\" (UID: \"f54eea24-bcc0-4c11-a687-351240b1adb8\") "
Oct 31 03:45:18.225736 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.225694    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fdef4c7-b0f9-4058-8629-700107f59688-kube-api-access-6zjwv\" (UniqueName: \"kubernetes.io/projected/3fdef4c7-b0f9-4058-8629-700107f59688-kube-api-access-6zjwv\") pod \"3fdef4c7-b0f9-4058-8629-700107f59688\" (UID: \"3fdef4c7-b0f9-4058-8629-700107f59688\") "
Oct 31 03:45:18.225792 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.225754    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54eea24-bcc0-4c11-a687-351240b1adb8-maas-api-service-ca" pod "f54eea24-bcc0-4c11-a687-351240b1adb8" (UID: "f54eea24-bcc0-4c11-a687-351240b1adb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:45:18.225792 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.225773    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fdef4c7-b0f9-4058-8629-700107f59688-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdef4c7-b0f9-4058-8629-700107f59688-maas-api-service-ca\") pod \"3fdef4c7-b0f9-4058-8629-700107f59688\" (UID: \"3fdef4c7-b0f9-4058-8629-700107f59688\") "
Oct 31 03:45:18.226213 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.226157    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fdef4c7-b0f9-4058-8629-700107f59688-maas-api-service-ca" pod "3fdef4c7-b0f9-4058-8629-700107f59688" (UID: "3fdef4c7-b0f9-4058-8629-700107f59688"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:45:18.226731 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.226710    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f54eea24-bcc0-4c11-a687-351240b1adb8-maas-api-service-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:45:18.226799 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.226737    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdef4c7-b0f9-4058-8629-700107f59688-maas-api-service-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:45:18.229036 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.229010    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54eea24-bcc0-4c11-a687-351240b1adb8-kube-api-access-qhcvg" pod "f54eea24-bcc0-4c11-a687-351240b1adb8" (UID: "f54eea24-bcc0-4c11-a687-351240b1adb8"). InnerVolumeSpecName "kube-api-access-qhcvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:45:18.229171 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.229151    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fdef4c7-b0f9-4058-8629-700107f59688-kube-api-access-6zjwv" pod "3fdef4c7-b0f9-4058-8629-700107f59688" (UID: "3fdef4c7-b0f9-4058-8629-700107f59688"). InnerVolumeSpecName "kube-api-access-6zjwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:45:18.328592 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.328515    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhcvg\" (UniqueName: \"kubernetes.io/projected/f54eea24-bcc0-4c11-a687-351240b1adb8-kube-api-access-qhcvg\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:45:18.328592 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.328539    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zjwv\" (UniqueName: \"kubernetes.io/projected/3fdef4c7-b0f9-4058-8629-700107f59688-kube-api-access-6zjwv\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:45:18.503461 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.503423    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-dqwsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 31 03:45:18.503730 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.503699    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm" podUID="84553d7f-51d4-4b72-b2dd-ee522e5413ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 31 03:45:18.746877 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.746842    4447 generic.go:356] "Generic (PLEG): container finished" podID="4d035a15-aefe-450f-8f99-bba3c3f2c15c" containerID="ac3db44c5f727d189b19b0fa18520d5479d9bff951aaebd58fce811333e320ff" exitCode=60
Oct 31 03:45:18.747248 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.746974    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p" event={"ID":"4d035a15-aefe-450f-8f99-bba3c3f2c15c","Type":"ContainerDied","Data":"ac3db44c5f727d189b19b0fa18520d5479d9bff951aaebd58fce811333e320ff"}
Oct 31 03:45:18.747248 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.747024    4447 scope.go:122] "RemoveContainer" containerID="7ba925f393dff527c450b5ada06ffa638b88852d58d49f016033c0903533f83a"
Oct 31 03:45:18.747875 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.747840    4447 scope.go:122] "RemoveContainer" containerID="ac3db44c5f727d189b19b0fa18520d5479d9bff951aaebd58fce811333e320ff"
Oct 31 03:45:18.750057 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.750040    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl"
Oct 31 03:45:18.750188 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:45:18.750156    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p_odh-ai-gateway-infra(4d035a15-aefe-450f-8f99-bba3c3f2c15c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p" podUID="4d035a15-aefe-450f-8f99-bba3c3f2c15c"
Oct 31 03:45:18.750274 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.750078    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl" event={"ID":"3fdef4c7-b0f9-4058-8629-700107f59688","Type":"ContainerDied","Data":"4dbc7aeca75adede43aa46afaa7f75285f98e65d2b7f15ec1b439411206405c8"}
Oct 31 03:45:18.752042 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.751746    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq" event={"ID":"f54eea24-bcc0-4c11-a687-351240b1adb8","Type":"ContainerDied","Data":"cdc80d6f7601fe2a45ad6f1b00e92a1c418315f88abf6a2d9c6e44804a44d534"}
Oct 31 03:45:18.752042 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.751768    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq"
Oct 31 03:45:18.755056 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.755025    4447 generic.go:356] "Generic (PLEG): container finished" podID="43d26fd6-e12c-4144-9b06-1144d5beb09c" containerID="462cb52258eebf825549d4895d530883957c49c47e1f25fa4b107c13bbf4eabc" exitCode=60
Oct 31 03:45:18.755161 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.755142    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz" event={"ID":"43d26fd6-e12c-4144-9b06-1144d5beb09c","Type":"ContainerDied","Data":"462cb52258eebf825549d4895d530883957c49c47e1f25fa4b107c13bbf4eabc"}
Oct 31 03:45:18.761422 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.761397    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn" event={"ID":"b0cd404b-1899-4e68-a8d1-09d37d8cb5e7","Type":"ContainerStarted","Data":"65d3dcb6fce444f0f0625af3012ea0259aeb86ef6a20d21e69d4670b895e534e"}
Oct 31 03:45:18.761526 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.761452    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn"
Oct 31 03:45:18.761621 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.761605    4447 scope.go:122] "RemoveContainer" containerID="462cb52258eebf825549d4895d530883957c49c47e1f25fa4b107c13bbf4eabc"
Oct 31 03:45:18.761920 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:45:18.761902    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz_odh-ai-gateway-infra(43d26fd6-e12c-4144-9b06-1144d5beb09c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz" podUID="43d26fd6-e12c-4144-9b06-1144d5beb09c"
Oct 31 03:45:18.764159 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.764144    4447 scope.go:122] "RemoveContainer" containerID="7cc9f39701acbde46cf01ef6a8a52cfcdfc516ba90846d7753257d62807ebeae"
Oct 31 03:45:18.808069 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.808046    4447 scope.go:122] "RemoveContainer" containerID="c1df8d846d44d1d9771f5839cc77b69d13adf1facf09e6c4f7d6e1059289b548"
Oct 31 03:45:18.820931 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.820906    4447 scope.go:122] "RemoveContainer" containerID="7e5b2bf983670522595af635935d5bef15628c0782289b5d5581784f4723622f"
Oct 31 03:45:18.868814 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:18.868762    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz"]
Oct 31 03:45:19.003483 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:19.003442    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq"]
Oct 31 03:45:19.008501 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:19.008475    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890305-2jkgq"]
Oct 31 03:45:19.030298 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:45:19.030263    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:45:19.232434 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:19.232385    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn" podStartSLOduration=3.364339856 podStartE2EDuration="4.232370121s" totalImagesPullingTime="868.030265ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:19.229451894 +0000 UTC m=+1544.740434272" watchObservedRunningTime="2026-10-31 03:45:19.232370121 +0000 UTC m=+1544.743352497"
Oct 31 03:45:19.250929 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:19.250870    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p"]
Oct 31 03:45:19.361714 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:19.361640    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl"]
Oct 31 03:45:19.366773 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:19.366747    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-14cb5a-29890305-4p7vl"]
Oct 31 03:45:19.506416 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:19.506375    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-dqwsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 31 03:45:19.506608 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:19.506444    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm" podUID="84553d7f-51d4-4b72-b2dd-ee522e5413ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 31 03:45:20.165259 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:20.163926    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz"
Oct 31 03:45:20.291917 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:20.284674    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p"
Oct 31 03:45:20.313949 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:20.313865    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43d26fd6-e12c-4144-9b06-1144d5beb09c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d26fd6-e12c-4144-9b06-1144d5beb09c-maas-api-service-ca\") pod \"43d26fd6-e12c-4144-9b06-1144d5beb09c\" (UID: \"43d26fd6-e12c-4144-9b06-1144d5beb09c\") "
Oct 31 03:45:20.313949 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:20.313936    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d26fd6-e12c-4144-9b06-1144d5beb09c-kube-api-access-hzbgp\" (UniqueName: \"kubernetes.io/projected/43d26fd6-e12c-4144-9b06-1144d5beb09c-kube-api-access-hzbgp\") pod \"43d26fd6-e12c-4144-9b06-1144d5beb09c\" (UID: \"43d26fd6-e12c-4144-9b06-1144d5beb09c\") "
Oct 31 03:45:20.314544 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:20.314521    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d26fd6-e12c-4144-9b06-1144d5beb09c-maas-api-service-ca" pod "43d26fd6-e12c-4144-9b06-1144d5beb09c" (UID: "43d26fd6-e12c-4144-9b06-1144d5beb09c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:45:20.314795 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:20.314778    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d26fd6-e12c-4144-9b06-1144d5beb09c-maas-api-service-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:45:20.316471 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:20.316449    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d26fd6-e12c-4144-9b06-1144d5beb09c-kube-api-access-hzbgp" pod "43d26fd6-e12c-4144-9b06-1144d5beb09c" (UID: "43d26fd6-e12c-4144-9b06-1144d5beb09c"). InnerVolumeSpecName "kube-api-access-hzbgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:45:20.416035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:20.415953    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d035a15-aefe-450f-8f99-bba3c3f2c15c-kube-api-access-5zsgv\" (UniqueName: \"kubernetes.io/projected/4d035a15-aefe-450f-8f99-bba3c3f2c15c-kube-api-access-5zsgv\") pod \"4d035a15-aefe-450f-8f99-bba3c3f2c15c\" (UID: \"4d035a15-aefe-450f-8f99-bba3c3f2c15c\") "
Oct 31 03:45:20.416191 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:20.416129    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d035a15-aefe-450f-8f99-bba3c3f2c15c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d035a15-aefe-450f-8f99-bba3c3f2c15c-maas-api-service-ca\") pod \"4d035a15-aefe-450f-8f99-bba3c3f2c15c\" (UID: \"4d035a15-aefe-450f-8f99-bba3c3f2c15c\") "
Oct 31 03:45:20.416504 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:20.416484    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d035a15-aefe-450f-8f99-bba3c3f2c15c-maas-api-service-ca" pod "4d035a15-aefe-450f-8f99-bba3c3f2c15c" (UID: "4d035a15-aefe-450f-8f99-bba3c3f2c15c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:45:20.416706 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:20.416690    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzbgp\" (UniqueName: \"kubernetes.io/projected/43d26fd6-e12c-4144-9b06-1144d5beb09c-kube-api-access-hzbgp\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:45:20.416742 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:20.416709    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d035a15-aefe-450f-8f99-bba3c3f2c15c-maas-api-service-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:45:20.418560 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:20.418542    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d035a15-aefe-450f-8f99-bba3c3f2c15c-kube-api-access-5zsgv" pod "4d035a15-aefe-450f-8f99-bba3c3f2c15c" (UID: "4d035a15-aefe-450f-8f99-bba3c3f2c15c"). InnerVolumeSpecName "kube-api-access-5zsgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:45:20.502908 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:20.502596    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-dqwsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 31 03:45:20.502908 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:20.502652    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm" podUID="84553d7f-51d4-4b72-b2dd-ee522e5413ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 31 03:45:20.518625 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:20.518590    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zsgv\" (UniqueName: \"kubernetes.io/projected/4d035a15-aefe-450f-8f99-bba3c3f2c15c-kube-api-access-5zsgv\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:45:20.652105 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:20.652068    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fdef4c7-b0f9-4058-8629-700107f59688" path="/var/lib/kubelet/pods/3fdef4c7-b0f9-4058-8629-700107f59688/volumes"
Oct 31 03:45:20.652833 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:20.652814    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f54eea24-bcc0-4c11-a687-351240b1adb8" path="/var/lib/kubelet/pods/f54eea24-bcc0-4c11-a687-351240b1adb8/volumes"
Oct 31 03:45:20.816951 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:20.816903    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz" event={"ID":"43d26fd6-e12c-4144-9b06-1144d5beb09c","Type":"ContainerDied","Data":"9af7368e2e9277aad9b5cee9a0334f40652938ae3021cca5f37ed7a13328a531"}
Oct 31 03:45:20.817143 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:20.816921    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz"
Oct 31 03:45:20.817143 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:20.817015    4447 scope.go:122] "RemoveContainer" containerID="462cb52258eebf825549d4895d530883957c49c47e1f25fa4b107c13bbf4eabc"
Oct 31 03:45:20.819085 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:20.818908    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p" event={"ID":"4d035a15-aefe-450f-8f99-bba3c3f2c15c","Type":"ContainerDied","Data":"fd55867ed4ab5cbec4903e8424e90a866d9bd597ea690ddaba600690bdd191d0"}
Oct 31 03:45:20.819085 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:20.818953    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p"
Oct 31 03:45:20.828995 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:20.828979    4447 scope.go:122] "RemoveContainer" containerID="ac3db44c5f727d189b19b0fa18520d5479d9bff951aaebd58fce811333e320ff"
Oct 31 03:45:21.004951 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.003968    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz"]
Oct 31 03:45:21.010479 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.010457    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-dec4a0-29890305-jpnnz"]
Oct 31 03:45:21.109130 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.109098    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p"]
Oct 31 03:45:21.114351 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.114328    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e00ad3-29890305-pv69p"]
Oct 31 03:45:21.158852 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.158825    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"]
Oct 31 03:45:21.159978 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.159958    4447 cpu_manager.go:414] "containerMap: removing container" podUID="3fdef4c7-b0f9-4058-8629-700107f59688" containerName="cleanup"
Oct 31 03:45:21.159978 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.159976    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fdef4c7-b0f9-4058-8629-700107f59688" containerName="cleanup"
Oct 31 03:45:21.160156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.159990    4447 cpu_manager.go:414] "containerMap: removing container" podUID="f54eea24-bcc0-4c11-a687-351240b1adb8" containerName="cleanup"
Oct 31 03:45:21.160156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.159995    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f54eea24-bcc0-4c11-a687-351240b1adb8" containerName="cleanup"
Oct 31 03:45:21.160156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160002    4447 cpu_manager.go:414] "containerMap: removing container" podUID="43d26fd6-e12c-4144-9b06-1144d5beb09c" containerName="cleanup"
Oct 31 03:45:21.160156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160006    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d26fd6-e12c-4144-9b06-1144d5beb09c" containerName="cleanup"
Oct 31 03:45:21.160156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160012    4447 cpu_manager.go:414] "containerMap: removing container" podUID="f54eea24-bcc0-4c11-a687-351240b1adb8" containerName="cleanup"
Oct 31 03:45:21.160156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160017    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f54eea24-bcc0-4c11-a687-351240b1adb8" containerName="cleanup"
Oct 31 03:45:21.160156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160036    4447 cpu_manager.go:414] "containerMap: removing container" podUID="3fdef4c7-b0f9-4058-8629-700107f59688" containerName="cleanup"
Oct 31 03:45:21.160156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160041    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fdef4c7-b0f9-4058-8629-700107f59688" containerName="cleanup"
Oct 31 03:45:21.160156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160055    4447 cpu_manager.go:414] "containerMap: removing container" podUID="4d035a15-aefe-450f-8f99-bba3c3f2c15c" containerName="cleanup"
Oct 31 03:45:21.160156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160060    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d035a15-aefe-450f-8f99-bba3c3f2c15c" containerName="cleanup"
Oct 31 03:45:21.160156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160067    4447 cpu_manager.go:414] "containerMap: removing container" podUID="4d035a15-aefe-450f-8f99-bba3c3f2c15c" containerName="cleanup"
Oct 31 03:45:21.160156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160072    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d035a15-aefe-450f-8f99-bba3c3f2c15c" containerName="cleanup"
Oct 31 03:45:21.160156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160080    4447 cpu_manager.go:414] "containerMap: removing container" podUID="4d035a15-aefe-450f-8f99-bba3c3f2c15c" containerName="cleanup"
Oct 31 03:45:21.160156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160084    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d035a15-aefe-450f-8f99-bba3c3f2c15c" containerName="cleanup"
Oct 31 03:45:21.160156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160108    4447 cpu_manager.go:414] "containerMap: removing container" podUID="43d26fd6-e12c-4144-9b06-1144d5beb09c" containerName="cleanup"
Oct 31 03:45:21.160156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160112    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d26fd6-e12c-4144-9b06-1144d5beb09c" containerName="cleanup"
Oct 31 03:45:21.160156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160120    4447 cpu_manager.go:414] "containerMap: removing container" podUID="3fdef4c7-b0f9-4058-8629-700107f59688" containerName="cleanup"
Oct 31 03:45:21.160156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160124    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fdef4c7-b0f9-4058-8629-700107f59688" containerName="cleanup"
Oct 31 03:45:21.160752 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160255    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="f54eea24-bcc0-4c11-a687-351240b1adb8" containerName="cleanup"
Oct 31 03:45:21.160752 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160264    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d26fd6-e12c-4144-9b06-1144d5beb09c" containerName="cleanup"
Oct 31 03:45:21.160752 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160271    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fdef4c7-b0f9-4058-8629-700107f59688" containerName="cleanup"
Oct 31 03:45:21.160752 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160276    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d035a15-aefe-450f-8f99-bba3c3f2c15c" containerName="cleanup"
Oct 31 03:45:21.160752 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160281    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d035a15-aefe-450f-8f99-bba3c3f2c15c" containerName="cleanup"
Oct 31 03:45:21.160752 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160287    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d035a15-aefe-450f-8f99-bba3c3f2c15c" containerName="cleanup"
Oct 31 03:45:21.160752 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160293    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d26fd6-e12c-4144-9b06-1144d5beb09c" containerName="cleanup"
Oct 31 03:45:21.160752 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160301    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fdef4c7-b0f9-4058-8629-700107f59688" containerName="cleanup"
Oct 31 03:45:21.160752 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.160307    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="f54eea24-bcc0-4c11-a687-351240b1adb8" containerName="cleanup"
Oct 31 03:45:21.165788 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.165770    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"
Oct 31 03:45:21.218785 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.218757    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-8de398-serving-cert\""
Oct 31 03:45:21.226665 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.226635    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"]
Oct 31 03:45:21.233013 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.230322    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r\" (UID: \"be8c55fa-3e54-42e4-86f6-fe5f702b5df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"
Oct 31 03:45:21.233013 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.230392    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssq7f\" (UniqueName: \"kubernetes.io/projected/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-kube-api-access-ssq7f\") pod \"maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r\" (UID: \"be8c55fa-3e54-42e4-86f6-fe5f702b5df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"
Oct 31 03:45:21.233013 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.230435    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r\" (UID: \"be8c55fa-3e54-42e4-86f6-fe5f702b5df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"
Oct 31 03:45:21.233013 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.230613    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r\" (UID: \"be8c55fa-3e54-42e4-86f6-fe5f702b5df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"
Oct 31 03:45:21.331580 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.331543    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r\" (UID: \"be8c55fa-3e54-42e4-86f6-fe5f702b5df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"
Oct 31 03:45:21.331777 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.331638    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r\" (UID: \"be8c55fa-3e54-42e4-86f6-fe5f702b5df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"
Oct 31 03:45:21.331777 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.331708    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssq7f\" (UniqueName: \"kubernetes.io/projected/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-kube-api-access-ssq7f\") pod \"maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r\" (UID: \"be8c55fa-3e54-42e4-86f6-fe5f702b5df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"
Oct 31 03:45:21.331777 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.331753    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r\" (UID: \"be8c55fa-3e54-42e4-86f6-fe5f702b5df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"
Oct 31 03:45:21.332009 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:45:21.331986    4447 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-serving-cert: secret "maas-api-e2e-shared-b-w5-8de398-serving-cert" not found
Oct 31 03:45:21.332059 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:45:21.332051    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-maas-api-tls podName:be8c55fa-3e54-42e4-86f6-fe5f702b5df9 nodeName:}" failed. No retries permitted until 2026-10-31 03:45:21.83203023 +0000 UTC m=+1547.343012605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-maas-api-tls") pod "maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r" (UID: "be8c55fa-3e54-42e4-86f6-fe5f702b5df9") : secret "maas-api-e2e-shared-b-w5-8de398-serving-cert" not found
Oct 31 03:45:21.333335 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.333283    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r\" (UID: \"be8c55fa-3e54-42e4-86f6-fe5f702b5df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"
Oct 31 03:45:21.336204 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.336179    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r\" (UID: \"be8c55fa-3e54-42e4-86f6-fe5f702b5df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"
Oct 31 03:45:21.407464 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.407378    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssq7f\" (UniqueName: \"kubernetes.io/projected/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-kube-api-access-ssq7f\") pod \"maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r\" (UID: \"be8c55fa-3e54-42e4-86f6-fe5f702b5df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"
Oct 31 03:45:21.507160 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.503090    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-dqwsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 31 03:45:21.507160 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.503159    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm" podUID="84553d7f-51d4-4b72-b2dd-ee522e5413ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 31 03:45:21.846641 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.846607    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r\" (UID: \"be8c55fa-3e54-42e4-86f6-fe5f702b5df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"
Oct 31 03:45:21.849314 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:21.849292    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r\" (UID: \"be8c55fa-3e54-42e4-86f6-fe5f702b5df9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"
Oct 31 03:45:22.104547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:22.104456    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"
Oct 31 03:45:22.492155 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:22.490839    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-dqwsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 31 03:45:22.492155 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:22.490914    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm" podUID="84553d7f-51d4-4b72-b2dd-ee522e5413ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 31 03:45:22.504939 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:22.497558    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"]
Oct 31 03:45:22.504939 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:45:22.501789    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8c55fa_3e54_42e4_86f6_fe5f702b5df9.slice/crio-835b915911159786501a931796efbd1b2890acdfc1609868c92b8875a93168a5 WatchSource:0}: Error finding container 835b915911159786501a931796efbd1b2890acdfc1609868c92b8875a93168a5: Status 404 returned error can't find the container with id 835b915911159786501a931796efbd1b2890acdfc1609868c92b8875a93168a5
Oct 31 03:45:22.615115 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:22.615076    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43d26fd6-e12c-4144-9b06-1144d5beb09c" path="/var/lib/kubelet/pods/43d26fd6-e12c-4144-9b06-1144d5beb09c/volumes"
Oct 31 03:45:22.615728 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:22.615709    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d035a15-aefe-450f-8f99-bba3c3f2c15c" path="/var/lib/kubelet/pods/4d035a15-aefe-450f-8f99-bba3c3f2c15c/volumes"
Oct 31 03:45:22.849191 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:22.849093    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r" event={"ID":"be8c55fa-3e54-42e4-86f6-fe5f702b5df9","Type":"ContainerStarted","Data":"835b915911159786501a931796efbd1b2890acdfc1609868c92b8875a93168a5"}
Oct 31 03:45:23.489000 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:23.488961    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-dqwsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 31 03:45:23.489168 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:23.489023    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm" podUID="84553d7f-51d4-4b72-b2dd-ee522e5413ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 31 03:45:23.863467 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:23.863430    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r" event={"ID":"be8c55fa-3e54-42e4-86f6-fe5f702b5df9","Type":"ContainerStarted","Data":"0cee3a7fa4e22c691df0679156bd95939c103be13ebe5bef29442d5107ce48fe"}
Oct 31 03:45:23.863923 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:23.863539    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"
Oct 31 03:45:23.984507 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:23.984432    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r" podStartSLOduration=2.125495505 podStartE2EDuration="2.984412288s" totalImagesPullingTime="858.916783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:23.982756848 +0000 UTC m=+1549.493739226" watchObservedRunningTime="2026-10-31 03:45:23.984412288 +0000 UTC m=+1549.495394672"
Oct 31 03:45:24.801777 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:24.801745    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn"
Oct 31 03:45:24.828428 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:24.828357    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:24.829256 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:24.829131    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:24.830918 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:24.830898    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm"
Oct 31 03:45:27.301878 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.301844    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9"]
Oct 31 03:45:27.303012 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.302980    4447 cpu_manager.go:414] "containerMap: removing container" podUID="43d26fd6-e12c-4144-9b06-1144d5beb09c" containerName="cleanup"
Oct 31 03:45:27.303012 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.303002    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d26fd6-e12c-4144-9b06-1144d5beb09c" containerName="cleanup"
Oct 31 03:45:27.303012 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.303012    4447 cpu_manager.go:414] "containerMap: removing container" podUID="f54eea24-bcc0-4c11-a687-351240b1adb8" containerName="cleanup"
Oct 31 03:45:27.303012 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.303017    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f54eea24-bcc0-4c11-a687-351240b1adb8" containerName="cleanup"
Oct 31 03:45:27.303417 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.303131    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d26fd6-e12c-4144-9b06-1144d5beb09c" containerName="cleanup"
Oct 31 03:45:27.303417 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.303144    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fdef4c7-b0f9-4058-8629-700107f59688" containerName="cleanup"
Oct 31 03:45:27.303417 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.303152    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="f54eea24-bcc0-4c11-a687-351240b1adb8" containerName="cleanup"
Oct 31 03:45:27.307235 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.307212    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9"
Oct 31 03:45:27.366021 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.365984    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9"]
Oct 31 03:45:27.408556 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.408522    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-e38e21-dockercfg-ljdx2\""
Oct 31 03:45:27.409222 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.408635    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-e38e21\""
Oct 31 03:45:27.459720 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.459194    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk5kj\" (UniqueName: \"kubernetes.io/projected/8f6f0c1f-5401-4d66-abbf-1cc388649280-kube-api-access-nk5kj\") pod \"payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9\" (UID: \"8f6f0c1f-5401-4d66-abbf-1cc388649280\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9"
Oct 31 03:45:27.459720 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.459329    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6f0c1f-5401-4d66-abbf-1cc388649280-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9\" (UID: \"8f6f0c1f-5401-4d66-abbf-1cc388649280\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9"
Oct 31 03:45:27.476385 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.476351    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw"]
Oct 31 03:45:27.484590 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.484544    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw"
Oct 31 03:45:27.532521 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.531102    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw"]
Oct 31 03:45:27.561191 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.561081    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk5kj\" (UniqueName: \"kubernetes.io/projected/8f6f0c1f-5401-4d66-abbf-1cc388649280-kube-api-access-nk5kj\") pod \"payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9\" (UID: \"8f6f0c1f-5401-4d66-abbf-1cc388649280\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9"
Oct 31 03:45:27.561191 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.561179    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6f0c1f-5401-4d66-abbf-1cc388649280-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9\" (UID: \"8f6f0c1f-5401-4d66-abbf-1cc388649280\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9"
Oct 31 03:45:27.561480 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.561238    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcjtn\" (UniqueName: \"kubernetes.io/projected/2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc-kube-api-access-qcjtn\") pod \"payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw\" (UID: \"2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw"
Oct 31 03:45:27.561480 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.561280    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw\" (UID: \"2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw"
Oct 31 03:45:27.562958 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.562332    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6f0c1f-5401-4d66-abbf-1cc388649280-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9\" (UID: \"8f6f0c1f-5401-4d66-abbf-1cc388649280\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9"
Oct 31 03:45:27.656712 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.656653    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk5kj\" (UniqueName: \"kubernetes.io/projected/8f6f0c1f-5401-4d66-abbf-1cc388649280-kube-api-access-nk5kj\") pod \"payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9\" (UID: \"8f6f0c1f-5401-4d66-abbf-1cc388649280\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9"
Oct 31 03:45:27.670506 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.669935    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcjtn\" (UniqueName: \"kubernetes.io/projected/2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc-kube-api-access-qcjtn\") pod \"payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw\" (UID: \"2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw"
Oct 31 03:45:27.670706 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.670564    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw\" (UID: \"2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw"
Oct 31 03:45:27.673004 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.672974    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw\" (UID: \"2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw"
Oct 31 03:45:27.763975 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.760567    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcjtn\" (UniqueName: \"kubernetes.io/projected/2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc-kube-api-access-qcjtn\") pod \"payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw\" (UID: \"2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw"
Oct 31 03:45:27.819463 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.819369    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw"
Oct 31 03:45:27.927935 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:27.927856    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9"
Oct 31 03:45:28.209337 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:45:28.209305    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4d1b66_e0c6_411e_9d68_59e9a1ca44dc.slice/crio-f349149cd14a05ba35a26b574fe587fca6b435949e6b84b3ceb57487ebdc96b5 WatchSource:0}: Error finding container f349149cd14a05ba35a26b574fe587fca6b435949e6b84b3ceb57487ebdc96b5: Status 404 returned error can't find the container with id f349149cd14a05ba35a26b574fe587fca6b435949e6b84b3ceb57487ebdc96b5
Oct 31 03:45:28.211228 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:28.211210    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw"]
Oct 31 03:45:28.327310 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:45:28.327283    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f6f0c1f_5401_4d66_abbf_1cc388649280.slice/crio-4563fff24432c7e5c2792c6f02b3839c5edd310458a4d4624d6367a6f158933a WatchSource:0}: Error finding container 4563fff24432c7e5c2792c6f02b3839c5edd310458a4d4624d6367a6f158933a: Status 404 returned error can't find the container with id 4563fff24432c7e5c2792c6f02b3839c5edd310458a4d4624d6367a6f158933a
Oct 31 03:45:28.327717 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:28.327530    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9"]
Oct 31 03:45:28.920701 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:28.920665    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9" event={"ID":"8f6f0c1f-5401-4d66-abbf-1cc388649280","Type":"ContainerStarted","Data":"230c1e51fc57aa1fbfa09557eddebf3f4cf7e0e074a27cda0c962b69a2b6d9af"}
Oct 31 03:45:28.920955 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:28.920730    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9" event={"ID":"8f6f0c1f-5401-4d66-abbf-1cc388649280","Type":"ContainerStarted","Data":"4563fff24432c7e5c2792c6f02b3839c5edd310458a4d4624d6367a6f158933a"}
Oct 31 03:45:28.920955 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:28.920757    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9"
Oct 31 03:45:28.922632 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:28.922606    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw" event={"ID":"2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc","Type":"ContainerStarted","Data":"6c69ceb9a4af7433376316fdd083fcde1af88ced01f3e6aa818f03be075bbae7"}
Oct 31 03:45:28.922725 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:28.922646    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw" event={"ID":"2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc","Type":"ContainerStarted","Data":"f349149cd14a05ba35a26b574fe587fca6b435949e6b84b3ceb57487ebdc96b5"}
Oct 31 03:45:28.922725 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:28.922665    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw"
Oct 31 03:45:29.039652 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:29.039603    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9" podStartSLOduration=2.039588637 podStartE2EDuration="2.039588637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:29.036949726 +0000 UTC m=+1554.547932102" watchObservedRunningTime="2026-10-31 03:45:29.039588637 +0000 UTC m=+1554.550571013"
Oct 31 03:45:29.159093 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:29.159043    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw" podStartSLOduration=2.159027417 podStartE2EDuration="2.159027417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:29.155054625 +0000 UTC m=+1554.666037004" watchObservedRunningTime="2026-10-31 03:45:29.159027417 +0000 UTC m=+1554.670009794"
Oct 31 03:45:29.870699 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:29.870667    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f"]
Oct 31 03:45:29.877643 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:29.877610    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f"
Oct 31 03:45:29.882994 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:29.882960    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"
Oct 31 03:45:29.954531 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:29.954489    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f"]
Oct 31 03:45:29.968459 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:29.959875    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjtx5\" (UniqueName: \"kubernetes.io/projected/1f0836ef-5542-45f6-8385-f71587717e3f-kube-api-access-hjtx5\") pod \"payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f\" (UID: \"1f0836ef-5542-45f6-8385-f71587717e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f"
Oct 31 03:45:29.968992 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:29.968957    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f0836ef-5542-45f6-8385-f71587717e3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f\" (UID: \"1f0836ef-5542-45f6-8385-f71587717e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f"
Oct 31 03:45:29.973752 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:29.973722    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-71186061\""
Oct 31 03:45:29.980423 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:29.973911    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-71186061-dockercfg-wsdr5\""
Oct 31 03:45:30.013599 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.013567    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk"]
Oct 31 03:45:30.019309 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.019286    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk"
Oct 31 03:45:30.070270 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.070236    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjtx5\" (UniqueName: \"kubernetes.io/projected/1f0836ef-5542-45f6-8385-f71587717e3f-kube-api-access-hjtx5\") pod \"payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f\" (UID: \"1f0836ef-5542-45f6-8385-f71587717e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f"
Oct 31 03:45:30.070489 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.070354    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95676bc2-e87a-4f52-8740-768b7f2a48c5-plugins-config-volume\") pod \"payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk\" (UID: \"95676bc2-e87a-4f52-8740-768b7f2a48c5\") " pod="openshift-ingress/payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk"
Oct 31 03:45:30.070489 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.070379    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f0836ef-5542-45f6-8385-f71587717e3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f\" (UID: \"1f0836ef-5542-45f6-8385-f71587717e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f"
Oct 31 03:45:30.071073 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.071043    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8n7j\" (UniqueName: \"kubernetes.io/projected/95676bc2-e87a-4f52-8740-768b7f2a48c5-kube-api-access-f8n7j\") pod \"payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk\" (UID: \"95676bc2-e87a-4f52-8740-768b7f2a48c5\") " pod="openshift-ingress/payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk"
Oct 31 03:45:30.072142 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.072112    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f0836ef-5542-45f6-8385-f71587717e3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f\" (UID: \"1f0836ef-5542-45f6-8385-f71587717e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f"
Oct 31 03:45:30.082305 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.081967    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk"]
Oct 31 03:45:30.163567 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.163478    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjtx5\" (UniqueName: \"kubernetes.io/projected/1f0836ef-5542-45f6-8385-f71587717e3f-kube-api-access-hjtx5\") pod \"payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f\" (UID: \"1f0836ef-5542-45f6-8385-f71587717e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f"
Oct 31 03:45:30.182066 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.180331    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95676bc2-e87a-4f52-8740-768b7f2a48c5-plugins-config-volume\") pod \"payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk\" (UID: \"95676bc2-e87a-4f52-8740-768b7f2a48c5\") " pod="openshift-ingress/payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk"
Oct 31 03:45:30.182066 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.180607    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8n7j\" (UniqueName: \"kubernetes.io/projected/95676bc2-e87a-4f52-8740-768b7f2a48c5-kube-api-access-f8n7j\") pod \"payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk\" (UID: \"95676bc2-e87a-4f52-8740-768b7f2a48c5\") " pod="openshift-ingress/payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk"
Oct 31 03:45:30.182990 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.182932    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95676bc2-e87a-4f52-8740-768b7f2a48c5-plugins-config-volume\") pod \"payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk\" (UID: \"95676bc2-e87a-4f52-8740-768b7f2a48c5\") " pod="openshift-ingress/payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk"
Oct 31 03:45:30.205154 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.205117    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"]
Oct 31 03:45:30.210156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.208823    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f"
Oct 31 03:45:30.212398 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.212378    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.240192 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.240165    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"]
Oct 31 03:45:30.328969 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.323743    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8n7j\" (UniqueName: \"kubernetes.io/projected/95676bc2-e87a-4f52-8740-768b7f2a48c5-kube-api-access-f8n7j\") pod \"payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk\" (UID: \"95676bc2-e87a-4f52-8740-768b7f2a48c5\") " pod="openshift-ingress/payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk"
Oct 31 03:45:30.328969 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.324791    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.328969 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.324844    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfsjr\" (UniqueName: \"kubernetes.io/projected/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-kube-api-access-zfsjr\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.328969 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.324873    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.328969 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.324939    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.328969 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.324959    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.328969 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.325082    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.328969 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.325221    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.328969 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.325322    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.328969 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.325385    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.346682 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.345505    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk"
Oct 31 03:45:30.433909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.431138    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.433909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.431298    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.433909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.431392    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.433909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.431448    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.433909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.431546    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.433909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.431588    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfsjr\" (UniqueName: \"kubernetes.io/projected/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-kube-api-access-zfsjr\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.433909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.431591    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.433909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.431617    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.433909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.431681    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.433909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.431709    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.433909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.431926    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.433909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.432540    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.433909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.433415    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.433909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.433499    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.437153 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.437127    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.448228 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.448190    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.500856 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.498526    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfsjr\" (UniqueName: \"kubernetes.io/projected/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-kube-api-access-zfsjr\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.504609 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.504580    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7daa66-ba76-4d1a-9de8-2a68928f22b7-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-lcwvs\" (UID: \"3d7daa66-ba76-4d1a-9de8-2a68928f22b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.683668 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:45:30.683636    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0836ef_5542_45f6_8385_f71587717e3f.slice/crio-cf32e0bceff3cea9b19d7abbf61dd1fea4294bf2024f910295d8624efe289cd9 WatchSource:0}: Error finding container cf32e0bceff3cea9b19d7abbf61dd1fea4294bf2024f910295d8624efe289cd9: Status 404 returned error can't find the container with id cf32e0bceff3cea9b19d7abbf61dd1fea4294bf2024f910295d8624efe289cd9
Oct 31 03:45:30.684337 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.684313    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f"]
Oct 31 03:45:30.693046 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.693022    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:30.790691 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.790662    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk"]
Oct 31 03:45:30.792455 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:45:30.792431    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95676bc2_e87a_4f52_8740_768b7f2a48c5.slice/crio-e0b59575c1ee5c847e6e0d2b03c8ef40b21824b865a4c0474c750d287a016098 WatchSource:0}: Error finding container e0b59575c1ee5c847e6e0d2b03c8ef40b21824b865a4c0474c750d287a016098: Status 404 returned error can't find the container with id e0b59575c1ee5c847e6e0d2b03c8ef40b21824b865a4c0474c750d287a016098
Oct 31 03:45:30.998025 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.997993    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk" event={"ID":"95676bc2-e87a-4f52-8740-768b7f2a48c5","Type":"ContainerStarted","Data":"6e6a0542579d6c7ee59f2743319208663a240502ec524c47ef9a543705ed7746"}
Oct 31 03:45:30.998420 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.998042    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk" event={"ID":"95676bc2-e87a-4f52-8740-768b7f2a48c5","Type":"ContainerStarted","Data":"e0b59575c1ee5c847e6e0d2b03c8ef40b21824b865a4c0474c750d287a016098"}
Oct 31 03:45:30.998420 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.998219    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk"
Oct 31 03:45:30.999801 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.999781    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f" event={"ID":"1f0836ef-5542-45f6-8385-f71587717e3f","Type":"ContainerStarted","Data":"c6e5b79f31bd003aa0c2ca78f126d946275defc1aad73fdf76d4c8a5f28020a4"}
Oct 31 03:45:30.999917 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.999818    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f" event={"ID":"1f0836ef-5542-45f6-8385-f71587717e3f","Type":"ContainerStarted","Data":"cf32e0bceff3cea9b19d7abbf61dd1fea4294bf2024f910295d8624efe289cd9"}
Oct 31 03:45:30.999917 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:30.999852    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f"
Oct 31 03:45:31.128526 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:31.128461    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk" podStartSLOduration=2.128445868 podStartE2EDuration="2.128445868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:31.12469353 +0000 UTC m=+1556.635675970" watchObservedRunningTime="2026-10-31 03:45:31.128445868 +0000 UTC m=+1556.639428273"
Oct 31 03:45:31.133770 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:45:31.133740    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7daa66_ba76_4d1a_9de8_2a68928f22b7.slice/crio-5ee2ad48710ffc50f4d9c1dcedaf7cce67a6e2304fea6a051589f1a5d230b916 WatchSource:0}: Error finding container 5ee2ad48710ffc50f4d9c1dcedaf7cce67a6e2304fea6a051589f1a5d230b916: Status 404 returned error can't find the container with id 5ee2ad48710ffc50f4d9c1dcedaf7cce67a6e2304fea6a051589f1a5d230b916
Oct 31 03:45:31.136087 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:31.136063    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"]
Oct 31 03:45:31.136645 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:31.136618    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:45:31.136693 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:31.136679    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:45:31.136728 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:31.136707    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:45:31.243771 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:31.242752    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f" podStartSLOduration=2.242611395 podStartE2EDuration="2.242611395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:31.2404004 +0000 UTC m=+1556.751382779" watchObservedRunningTime="2026-10-31 03:45:31.242611395 +0000 UTC m=+1556.753593773"
Oct 31 03:45:32.008956 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:32.008921    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs" event={"ID":"3d7daa66-ba76-4d1a-9de8-2a68928f22b7","Type":"ContainerStarted","Data":"0a63971e408ae6712ac263e2cc8e377a6c91501ccf9974bad9b31f76ada4e3fb"}
Oct 31 03:45:32.009343 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:32.008981    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs" event={"ID":"3d7daa66-ba76-4d1a-9de8-2a68928f22b7","Type":"ContainerStarted","Data":"5ee2ad48710ffc50f4d9c1dcedaf7cce67a6e2304fea6a051589f1a5d230b916"}
Oct 31 03:45:32.134607 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:32.134551    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs" podStartSLOduration=2.134536088 podStartE2EDuration="2.134536088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:32.128970713 +0000 UTC m=+1557.639953090" watchObservedRunningTime="2026-10-31 03:45:32.134536088 +0000 UTC m=+1557.645518464"
Oct 31 03:45:32.700221 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:32.700187    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:32.701452 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:32.701427    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-lcwvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 31 03:45:32.701604 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:32.701476    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs" podUID="3d7daa66-ba76-4d1a-9de8-2a68928f22b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 31 03:45:33.696909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:33.695475    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-lcwvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 31 03:45:33.696909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:33.695533    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs" podUID="3d7daa66-ba76-4d1a-9de8-2a68928f22b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 31 03:45:34.700360 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:34.700289    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-lcwvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 31 03:45:34.700360 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:34.700353    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs" podUID="3d7daa66-ba76-4d1a-9de8-2a68928f22b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 31 03:45:35.696912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:35.693841    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-lcwvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 31 03:45:35.696912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:35.693929    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs" podUID="3d7daa66-ba76-4d1a-9de8-2a68928f22b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 31 03:45:36.707011 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:36.706974    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-lcwvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 31 03:45:36.707597 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:36.707556    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs" podUID="3d7daa66-ba76-4d1a-9de8-2a68928f22b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 31 03:45:37.693462 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:37.693416    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-lcwvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 31 03:45:37.693673 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:37.693474    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs" podUID="3d7daa66-ba76-4d1a-9de8-2a68928f22b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 31 03:45:38.694272 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:38.694237    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-lcwvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 31 03:45:38.694793 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:38.694296    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs" podUID="3d7daa66-ba76-4d1a-9de8-2a68928f22b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 31 03:45:39.698380 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:39.698350    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:39.698794 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:39.698745    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:39.699504 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:39.699478    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs"
Oct 31 03:45:39.993809 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:39.993777    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw"
Oct 31 03:45:39.994375 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:39.994340    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9"
Oct 31 03:45:42.012292 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:42.012260    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk"
Oct 31 03:45:42.012661 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:42.012325    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f"
Oct 31 03:45:43.061442 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:43.061406    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-607bc420-bhtbg"]
Oct 31 03:45:43.111012 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:43.110976    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-607bc420-bhtbg"]
Oct 31 03:45:44.611136 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:45:44.611105    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13313afc-657b-4a56-a50d-e0547d6781de" path="/var/lib/kubelet/pods/13313afc-657b-4a56-a50d-e0547d6781de/volumes"
Oct 31 03:45:49.004390 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:45:49.004306    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:46:15.958871 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:15.958836    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"]
Oct 31 03:46:15.963509 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:15.963492    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"
Oct 31 03:46:16.018654 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:16.018608    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-e38e21-serving-cert\""
Oct 31 03:46:16.073393 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:16.073358    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"]
Oct 31 03:46:16.126457 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:16.126424    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49240af-df66-44be-97a9-8576b042437a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk\" (UID: \"b49240af-df66-44be-97a9-8576b042437a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"
Oct 31 03:46:16.126699 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:16.126675    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xh9k\" (UniqueName: \"kubernetes.io/projected/b49240af-df66-44be-97a9-8576b042437a-kube-api-access-9xh9k\") pod \"maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk\" (UID: \"b49240af-df66-44be-97a9-8576b042437a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"
Oct 31 03:46:16.126804 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:16.126788    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b49240af-df66-44be-97a9-8576b042437a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk\" (UID: \"b49240af-df66-44be-97a9-8576b042437a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"
Oct 31 03:46:16.126970 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:16.126947    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49240af-df66-44be-97a9-8576b042437a-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk\" (UID: \"b49240af-df66-44be-97a9-8576b042437a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"
Oct 31 03:46:16.228928 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:16.228821    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49240af-df66-44be-97a9-8576b042437a-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk\" (UID: \"b49240af-df66-44be-97a9-8576b042437a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"
Oct 31 03:46:16.229113 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:16.228971    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49240af-df66-44be-97a9-8576b042437a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk\" (UID: \"b49240af-df66-44be-97a9-8576b042437a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"
Oct 31 03:46:16.229186 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:16.229168    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xh9k\" (UniqueName: \"kubernetes.io/projected/b49240af-df66-44be-97a9-8576b042437a-kube-api-access-9xh9k\") pod \"maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk\" (UID: \"b49240af-df66-44be-97a9-8576b042437a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"
Oct 31 03:46:16.229409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:16.229380    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b49240af-df66-44be-97a9-8576b042437a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk\" (UID: \"b49240af-df66-44be-97a9-8576b042437a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"
Oct 31 03:46:16.230322 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:16.230293    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49240af-df66-44be-97a9-8576b042437a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk\" (UID: \"b49240af-df66-44be-97a9-8576b042437a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"
Oct 31 03:46:16.230467 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:46:16.229488    4447 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-serving-cert: secret "maas-api-e2e-shared-a-w1-e38e21-serving-cert" not found
Oct 31 03:46:16.230467 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:46:16.230400    4447 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b49240af-df66-44be-97a9-8576b042437a-maas-api-tls podName:b49240af-df66-44be-97a9-8576b042437a nodeName:}" failed. No retries permitted until 2026-10-31 03:46:16.730378011 +0000 UTC m=+1602.241360380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b49240af-df66-44be-97a9-8576b042437a-maas-api-tls") pod "maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk" (UID: "b49240af-df66-44be-97a9-8576b042437a") : secret "maas-api-e2e-shared-a-w1-e38e21-serving-cert" not found
Oct 31 03:46:16.232116 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:16.232093    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49240af-df66-44be-97a9-8576b042437a-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk\" (UID: \"b49240af-df66-44be-97a9-8576b042437a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"
Oct 31 03:46:16.292047 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:16.292020    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xh9k\" (UniqueName: \"kubernetes.io/projected/b49240af-df66-44be-97a9-8576b042437a-kube-api-access-9xh9k\") pod \"maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk\" (UID: \"b49240af-df66-44be-97a9-8576b042437a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"
Oct 31 03:46:16.739359 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:16.739324    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b49240af-df66-44be-97a9-8576b042437a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk\" (UID: \"b49240af-df66-44be-97a9-8576b042437a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"
Oct 31 03:46:16.741993 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:16.741969    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b49240af-df66-44be-97a9-8576b042437a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk\" (UID: \"b49240af-df66-44be-97a9-8576b042437a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"
Oct 31 03:46:16.878857 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:16.878808    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"
Oct 31 03:46:17.052313 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:17.052225    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5165bf8-jb2d9"]
Oct 31 03:46:17.102067 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:17.102036    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5165bf8-jb2d9"]
Oct 31 03:46:17.315470 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:46:17.315444    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49240af_df66_44be_97a9_8576b042437a.slice/crio-4e3964823719c0b1edb8ffd0aeac74d8ca009ff8b60350363008f0574f803c54 WatchSource:0}: Error finding container 4e3964823719c0b1edb8ffd0aeac74d8ca009ff8b60350363008f0574f803c54: Status 404 returned error can't find the container with id 4e3964823719c0b1edb8ffd0aeac74d8ca009ff8b60350363008f0574f803c54
Oct 31 03:46:17.315920 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:17.315899    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"]
Oct 31 03:46:17.317264 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:17.317248    4447 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:46:17.457794 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:17.457759    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk" event={"ID":"b49240af-df66-44be-97a9-8576b042437a","Type":"ContainerStarted","Data":"4e3964823719c0b1edb8ffd0aeac74d8ca009ff8b60350363008f0574f803c54"}
Oct 31 03:46:18.468124 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:18.468084    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk" event={"ID":"b49240af-df66-44be-97a9-8576b042437a","Type":"ContainerStarted","Data":"c2c709ea655d8d4112be47041ca6b34da98dd2c9b9082c2d10a5bdb12e6acc66"}
Oct 31 03:46:18.468565 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:18.468336    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"
Oct 31 03:46:18.585714 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:18.585660    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk" podStartSLOduration=2.715201324 podStartE2EDuration="3.585645025s" totalImagesPullingTime="870.443701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:46:18.585403773 +0000 UTC m=+1604.096386149" watchObservedRunningTime="2026-10-31 03:46:18.585645025 +0000 UTC m=+1604.096627402"
Oct 31 03:46:18.611356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:18.611319    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ade9412-157f-4df2-8928-117d91e4480c" path="/var/lib/kubelet/pods/1ade9412-157f-4df2-8928-117d91e4480c/volumes"
Oct 31 03:46:19.005224 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:46:19.005180    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:46:24.483837 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:24.483808    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"
Oct 31 03:46:39.543783 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:39.543746    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb"]
Oct 31 03:46:43.850810 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:43.850769    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc"]
Oct 31 03:46:43.858458 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:43.858430    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc"
Oct 31 03:46:43.961333 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:43.960040    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1180d818\""
Oct 31 03:46:43.961333 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:43.960233    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1180d818-dockercfg-46gbh\""
Oct 31 03:46:43.964161 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:43.964123    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc"]
Oct 31 03:46:44.000974 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:43.996480    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0963d8e4-63ff-42d7-a0d8-f7c1bcc73761-plugins-config-volume\") pod \"payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc\" (UID: \"0963d8e4-63ff-42d7-a0d8-f7c1bcc73761\") " pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc"
Oct 31 03:46:44.000974 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:43.996707    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jdfn\" (UniqueName: \"kubernetes.io/projected/0963d8e4-63ff-42d7-a0d8-f7c1bcc73761-kube-api-access-5jdfn\") pod \"payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc\" (UID: \"0963d8e4-63ff-42d7-a0d8-f7c1bcc73761\") " pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc"
Oct 31 03:46:44.098954 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:44.098864    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jdfn\" (UniqueName: \"kubernetes.io/projected/0963d8e4-63ff-42d7-a0d8-f7c1bcc73761-kube-api-access-5jdfn\") pod \"payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc\" (UID: \"0963d8e4-63ff-42d7-a0d8-f7c1bcc73761\") " pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc"
Oct 31 03:46:44.099454 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:44.099420    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0963d8e4-63ff-42d7-a0d8-f7c1bcc73761-plugins-config-volume\") pod \"payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc\" (UID: \"0963d8e4-63ff-42d7-a0d8-f7c1bcc73761\") " pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc"
Oct 31 03:46:44.100223 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:44.100205    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0963d8e4-63ff-42d7-a0d8-f7c1bcc73761-plugins-config-volume\") pod \"payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc\" (UID: \"0963d8e4-63ff-42d7-a0d8-f7c1bcc73761\") " pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc"
Oct 31 03:46:44.160642 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:44.160564    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jdfn\" (UniqueName: \"kubernetes.io/projected/0963d8e4-63ff-42d7-a0d8-f7c1bcc73761-kube-api-access-5jdfn\") pod \"payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc\" (UID: \"0963d8e4-63ff-42d7-a0d8-f7c1bcc73761\") " pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc"
Oct 31 03:46:44.177875 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:44.177503    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc"
Oct 31 03:46:44.544348 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:44.544305    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb" podUID="b37925d4-a955-4e16-a0bc-2e9e98ae3d60" containerName="payload-pre-processing" containerID="cri-o://79e917607a0253514b18e20cd3a1e36a6af192f2d4e4c29a677b3933a7cd21ad" gracePeriod=25
Oct 31 03:46:44.592523 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:44.592495    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc"]
Oct 31 03:46:44.617877 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:46:44.617849    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0963d8e4_63ff_42d7_a0d8_f7c1bcc73761.slice/crio-3c102a11727d003022a6efe8598dcb7ca6bfc9635d3d6aaaeecbec1e66c8c37f WatchSource:0}: Error finding container 3c102a11727d003022a6efe8598dcb7ca6bfc9635d3d6aaaeecbec1e66c8c37f: Status 404 returned error can't find the container with id 3c102a11727d003022a6efe8598dcb7ca6bfc9635d3d6aaaeecbec1e66c8c37f
Oct 31 03:46:44.735362 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:44.735320    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc" event={"ID":"0963d8e4-63ff-42d7-a0d8-f7c1bcc73761","Type":"ContainerStarted","Data":"3c102a11727d003022a6efe8598dcb7ca6bfc9635d3d6aaaeecbec1e66c8c37f"}
Oct 31 03:46:44.737665 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:44.737641    4447 generic.go:356] "Generic (PLEG): container finished" podID="b37925d4-a955-4e16-a0bc-2e9e98ae3d60" containerID="79e917607a0253514b18e20cd3a1e36a6af192f2d4e4c29a677b3933a7cd21ad" exitCode=0
Oct 31 03:46:44.737876 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:44.737852    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb" event={"ID":"b37925d4-a955-4e16-a0bc-2e9e98ae3d60","Type":"ContainerDied","Data":"79e917607a0253514b18e20cd3a1e36a6af192f2d4e4c29a677b3933a7cd21ad"}
Oct 31 03:46:44.815666 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:44.815618    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb"
Oct 31 03:46:44.914255 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:44.914226    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37925d4-a955-4e16-a0bc-2e9e98ae3d60-kube-api-access-m6bfm\" (UniqueName: \"kubernetes.io/projected/b37925d4-a955-4e16-a0bc-2e9e98ae3d60-kube-api-access-m6bfm\") pod \"b37925d4-a955-4e16-a0bc-2e9e98ae3d60\" (UID: \"b37925d4-a955-4e16-a0bc-2e9e98ae3d60\") "
Oct 31 03:46:44.914736 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:44.914307    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b37925d4-a955-4e16-a0bc-2e9e98ae3d60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37925d4-a955-4e16-a0bc-2e9e98ae3d60-plugins-config-volume\") pod \"b37925d4-a955-4e16-a0bc-2e9e98ae3d60\" (UID: \"b37925d4-a955-4e16-a0bc-2e9e98ae3d60\") "
Oct 31 03:46:44.914803 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:44.914739    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b37925d4-a955-4e16-a0bc-2e9e98ae3d60-plugins-config-volume" pod "b37925d4-a955-4e16-a0bc-2e9e98ae3d60" (UID: "b37925d4-a955-4e16-a0bc-2e9e98ae3d60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:46:44.915066 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:44.915049    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37925d4-a955-4e16-a0bc-2e9e98ae3d60-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:46:44.916770 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:44.916747    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37925d4-a955-4e16-a0bc-2e9e98ae3d60-kube-api-access-m6bfm" pod "b37925d4-a955-4e16-a0bc-2e9e98ae3d60" (UID: "b37925d4-a955-4e16-a0bc-2e9e98ae3d60"). InnerVolumeSpecName "kube-api-access-m6bfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:45.017041 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:45.017007    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6bfm\" (UniqueName: \"kubernetes.io/projected/b37925d4-a955-4e16-a0bc-2e9e98ae3d60-kube-api-access-m6bfm\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:46:45.748236 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:45.748209    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb"
Oct 31 03:46:45.748433 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:45.748254    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb" event={"ID":"b37925d4-a955-4e16-a0bc-2e9e98ae3d60","Type":"ContainerDied","Data":"aab2c20d9d5f2722472827b8b8f797b3e1255f6a82866eff2f1c2a679d0c657c"}
Oct 31 03:46:45.748433 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:45.748315    4447 scope.go:122] "RemoveContainer" containerID="79e917607a0253514b18e20cd3a1e36a6af192f2d4e4c29a677b3933a7cd21ad"
Oct 31 03:46:45.750351 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:45.750322    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc" event={"ID":"0963d8e4-63ff-42d7-a0d8-f7c1bcc73761","Type":"ContainerStarted","Data":"f316dc6c64dda0098a78e9a04b25114eb6e928eb76d6de5d0e2d253c1c10b318"}
Oct 31 03:46:45.750485 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:45.750456    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc"
Oct 31 03:46:45.915595 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:45.915561    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb"]
Oct 31 03:46:45.922868 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:45.922840    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bb362142-67d9489dcf-pqjmb"]
Oct 31 03:46:46.035848 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:46.035744    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc" podStartSLOduration=3.035726653 podStartE2EDuration="3.035726653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:46:46.03263635 +0000 UTC m=+1631.543618727" watchObservedRunningTime="2026-10-31 03:46:46.035726653 +0000 UTC m=+1631.546709033"
Oct 31 03:46:46.613023 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:46.612992    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b37925d4-a955-4e16-a0bc-2e9e98ae3d60" path="/var/lib/kubelet/pods/b37925d4-a955-4e16-a0bc-2e9e98ae3d60/volumes"
Oct 31 03:46:49.100491 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:46:49.100452    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:46:56.762636 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:46:56.762605    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc"
Oct 31 03:47:00.024617 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:00.024581    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"]
Oct 31 03:47:00.025072 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:00.024774    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld" podUID="01a1ebb8-9905-470d-bace-caf42a18af34" containerName="istio-proxy" containerID="cri-o://c394ffc0da9985e3da64bda2a13304450aef3fefe0708c3adb8857cac0b9bcb8" gracePeriod=30
Oct 31 03:47:05.324740 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.324699    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:47:05.499451 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.499416    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a1ebb8-9905-470d-bace-caf42a18af34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a1ebb8-9905-470d-bace-caf42a18af34-istiod-ca-cert\") pod \"01a1ebb8-9905-470d-bace-caf42a18af34\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") "
Oct 31 03:47:05.499659 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.499504    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a1ebb8-9905-470d-bace-caf42a18af34-istio-token\" (UniqueName: \"kubernetes.io/projected/01a1ebb8-9905-470d-bace-caf42a18af34-istio-token\") pod \"01a1ebb8-9905-470d-bace-caf42a18af34\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") "
Oct 31 03:47:05.499659 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.499535    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-istio-envoy\") pod \"01a1ebb8-9905-470d-bace-caf42a18af34\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") "
Oct 31 03:47:05.499770 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.499684    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01a1ebb8-9905-470d-bace-caf42a18af34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a1ebb8-9905-470d-bace-caf42a18af34-istio-podinfo\") pod \"01a1ebb8-9905-470d-bace-caf42a18af34\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") "
Oct 31 03:47:05.499770 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.499721    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a1ebb8-9905-470d-bace-caf42a18af34-kube-api-access-92992\" (UniqueName: \"kubernetes.io/projected/01a1ebb8-9905-470d-bace-caf42a18af34-kube-api-access-92992\") pod \"01a1ebb8-9905-470d-bace-caf42a18af34\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") "
Oct 31 03:47:05.499770 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.499729    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a1ebb8-9905-470d-bace-caf42a18af34-istiod-ca-cert" pod "01a1ebb8-9905-470d-bace-caf42a18af34" (UID: "01a1ebb8-9905-470d-bace-caf42a18af34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:47:05.499770 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.499747    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-workload-socket\") pod \"01a1ebb8-9905-470d-bace-caf42a18af34\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") "
Oct 31 03:47:05.500018 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.499802    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-workload-certs\") pod \"01a1ebb8-9905-470d-bace-caf42a18af34\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") "
Oct 31 03:47:05.500018 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.500009    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-credential-socket\") pod \"01a1ebb8-9905-470d-bace-caf42a18af34\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") "
Oct 31 03:47:05.500126 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.500049    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-istio-data\") pod \"01a1ebb8-9905-470d-bace-caf42a18af34\" (UID: \"01a1ebb8-9905-470d-bace-caf42a18af34\") "
Oct 31 03:47:05.501094 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.501067    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a1ebb8-9905-470d-bace-caf42a18af34-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:47:05.501837 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.501538    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-istio-data" pod "01a1ebb8-9905-470d-bace-caf42a18af34" (UID: "01a1ebb8-9905-470d-bace-caf42a18af34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:05.501837 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.501758    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-credential-socket" pod "01a1ebb8-9905-470d-bace-caf42a18af34" (UID: "01a1ebb8-9905-470d-bace-caf42a18af34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:05.504229 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.503362    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-workload-certs" pod "01a1ebb8-9905-470d-bace-caf42a18af34" (UID: "01a1ebb8-9905-470d-bace-caf42a18af34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:05.504229 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.503560    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-workload-socket" pod "01a1ebb8-9905-470d-bace-caf42a18af34" (UID: "01a1ebb8-9905-470d-bace-caf42a18af34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:05.504229 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.504163    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a1ebb8-9905-470d-bace-caf42a18af34-kube-api-access-92992" pod "01a1ebb8-9905-470d-bace-caf42a18af34" (UID: "01a1ebb8-9905-470d-bace-caf42a18af34"). InnerVolumeSpecName "kube-api-access-92992". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:05.504561 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.504536    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-istio-envoy" pod "01a1ebb8-9905-470d-bace-caf42a18af34" (UID: "01a1ebb8-9905-470d-bace-caf42a18af34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:05.505307 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.505276    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a1ebb8-9905-470d-bace-caf42a18af34-istio-token" pod "01a1ebb8-9905-470d-bace-caf42a18af34" (UID: "01a1ebb8-9905-470d-bace-caf42a18af34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:05.510132 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.510100    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01a1ebb8-9905-470d-bace-caf42a18af34-istio-podinfo" pod "01a1ebb8-9905-470d-bace-caf42a18af34" (UID: "01a1ebb8-9905-470d-bace-caf42a18af34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:47:05.602372 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.602283    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a1ebb8-9905-470d-bace-caf42a18af34-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:47:05.602372 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.602320    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92992\" (UniqueName: \"kubernetes.io/projected/01a1ebb8-9905-470d-bace-caf42a18af34-kube-api-access-92992\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:47:05.602372 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.602337    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:47:05.602372 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.602346    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:47:05.602372 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.602357    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:47:05.602372 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.602371    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:47:05.602695 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.602386    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a1ebb8-9905-470d-bace-caf42a18af34-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:47:05.602695 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:05.602402    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a1ebb8-9905-470d-bace-caf42a18af34-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:47:06.002764 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:06.002722    4447 generic.go:356] "Generic (PLEG): container finished" podID="01a1ebb8-9905-470d-bace-caf42a18af34" containerID="c394ffc0da9985e3da64bda2a13304450aef3fefe0708c3adb8857cac0b9bcb8" exitCode=0
Oct 31 03:47:06.003001 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:06.002846    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"
Oct 31 03:47:06.003001 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:06.002853    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld" event={"ID":"01a1ebb8-9905-470d-bace-caf42a18af34","Type":"ContainerDied","Data":"c394ffc0da9985e3da64bda2a13304450aef3fefe0708c3adb8857cac0b9bcb8"}
Oct 31 03:47:06.003001 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:06.002949    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld" event={"ID":"01a1ebb8-9905-470d-bace-caf42a18af34","Type":"ContainerDied","Data":"458f05c3ba098a0b6da22a09450738955c5420cd92b4864ab6a68c6ec863da94"}
Oct 31 03:47:06.003001 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:06.002971    4447 scope.go:122] "RemoveContainer" containerID="c394ffc0da9985e3da64bda2a13304450aef3fefe0708c3adb8857cac0b9bcb8"
Oct 31 03:47:06.016286 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:06.016265    4447 scope.go:122] "RemoveContainer" containerID="c394ffc0da9985e3da64bda2a13304450aef3fefe0708c3adb8857cac0b9bcb8"
Oct 31 03:47:06.016637 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:47:06.016621    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c394ffc0da9985e3da64bda2a13304450aef3fefe0708c3adb8857cac0b9bcb8\": container with ID starting with c394ffc0da9985e3da64bda2a13304450aef3fefe0708c3adb8857cac0b9bcb8 not found: ID does not exist" containerID="c394ffc0da9985e3da64bda2a13304450aef3fefe0708c3adb8857cac0b9bcb8"
Oct 31 03:47:06.016675 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:06.016648    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c394ffc0da9985e3da64bda2a13304450aef3fefe0708c3adb8857cac0b9bcb8"} err="failed to get container status \"c394ffc0da9985e3da64bda2a13304450aef3fefe0708c3adb8857cac0b9bcb8\": rpc error: code = NotFound desc = could not find container \"c394ffc0da9985e3da64bda2a13304450aef3fefe0708c3adb8857cac0b9bcb8\": container with ID starting with c394ffc0da9985e3da64bda2a13304450aef3fefe0708c3adb8857cac0b9bcb8 not found: ID does not exist"
Oct 31 03:47:06.172350 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:06.172318    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"]
Oct 31 03:47:06.222633 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:06.222601    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-909365-openshift-default-cc45f8c9c-lzrld"]
Oct 31 03:47:06.613200 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:06.613168    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a1ebb8-9905-470d-bace-caf42a18af34" path="/var/lib/kubelet/pods/01a1ebb8-9905-470d-bace-caf42a18af34/volumes"
Oct 31 03:47:15.534578 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:15.534544    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn"]
Oct 31 03:47:15.535065 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:15.534736    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn" podUID="b0cd404b-1899-4e68-a8d1-09d37d8cb5e7" containerName="maas-api" containerID="cri-o://65d3dcb6fce444f0f0625af3012ea0259aeb86ef6a20d21e69d4670b895e534e" gracePeriod=30
Oct 31 03:47:15.889279 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:15.889250    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn"
Oct 31 03:47:15.980573 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:15.980541    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-metrics-certs\") pod \"b0cd404b-1899-4e68-a8d1-09d37d8cb5e7\" (UID: \"b0cd404b-1899-4e68-a8d1-09d37d8cb5e7\") "
Oct 31 03:47:15.980740 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:15.980685    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-openshift-service-ca\") pod \"b0cd404b-1899-4e68-a8d1-09d37d8cb5e7\" (UID: \"b0cd404b-1899-4e68-a8d1-09d37d8cb5e7\") "
Oct 31 03:47:15.980789 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:15.980766    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-kube-api-access-2qh5j\" (UniqueName: \"kubernetes.io/projected/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-kube-api-access-2qh5j\") pod \"b0cd404b-1899-4e68-a8d1-09d37d8cb5e7\" (UID: \"b0cd404b-1899-4e68-a8d1-09d37d8cb5e7\") "
Oct 31 03:47:15.980923 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:15.980867    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-maas-api-tls\") pod \"b0cd404b-1899-4e68-a8d1-09d37d8cb5e7\" (UID: \"b0cd404b-1899-4e68-a8d1-09d37d8cb5e7\") "
Oct 31 03:47:15.981089 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:15.981062    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-openshift-service-ca" pod "b0cd404b-1899-4e68-a8d1-09d37d8cb5e7" (UID: "b0cd404b-1899-4e68-a8d1-09d37d8cb5e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:47:15.981603 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:15.981582    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-openshift-service-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:47:15.983098 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:15.983069    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-kube-api-access-2qh5j" pod "b0cd404b-1899-4e68-a8d1-09d37d8cb5e7" (UID: "b0cd404b-1899-4e68-a8d1-09d37d8cb5e7"). InnerVolumeSpecName "kube-api-access-2qh5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:15.983172 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:15.983102    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-maas-api-tls" pod "b0cd404b-1899-4e68-a8d1-09d37d8cb5e7" (UID: "b0cd404b-1899-4e68-a8d1-09d37d8cb5e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:47:15.983222 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:15.983175    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-metrics-certs" pod "b0cd404b-1899-4e68-a8d1-09d37d8cb5e7" (UID: "b0cd404b-1899-4e68-a8d1-09d37d8cb5e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:47:16.084922 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:16.084873    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-metrics-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:47:16.084922 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:16.084922    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qh5j\" (UniqueName: \"kubernetes.io/projected/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-kube-api-access-2qh5j\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:47:16.085187 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:16.084938    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7-maas-api-tls\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:47:16.107977 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:16.099455    4447 generic.go:356] "Generic (PLEG): container finished" podID="b0cd404b-1899-4e68-a8d1-09d37d8cb5e7" containerID="65d3dcb6fce444f0f0625af3012ea0259aeb86ef6a20d21e69d4670b895e534e" exitCode=0
Oct 31 03:47:16.107977 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:16.099804    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn" event={"ID":"b0cd404b-1899-4e68-a8d1-09d37d8cb5e7","Type":"ContainerDied","Data":"65d3dcb6fce444f0f0625af3012ea0259aeb86ef6a20d21e69d4670b895e534e"}
Oct 31 03:47:16.107977 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:16.099858    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn" event={"ID":"b0cd404b-1899-4e68-a8d1-09d37d8cb5e7","Type":"ContainerDied","Data":"809b3af5bf0ddeca02c3742679c99e742ffac39046a63b90ab50dde7ced1d303"}
Oct 31 03:47:16.107977 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:16.099906    4447 scope.go:122] "RemoveContainer" containerID="65d3dcb6fce444f0f0625af3012ea0259aeb86ef6a20d21e69d4670b895e534e"
Oct 31 03:47:16.107977 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:16.100082    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn"
Oct 31 03:47:16.132431 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:16.132405    4447 scope.go:122] "RemoveContainer" containerID="65d3dcb6fce444f0f0625af3012ea0259aeb86ef6a20d21e69d4670b895e534e"
Oct 31 03:47:16.132772 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:47:16.132750    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65d3dcb6fce444f0f0625af3012ea0259aeb86ef6a20d21e69d4670b895e534e\": container with ID starting with 65d3dcb6fce444f0f0625af3012ea0259aeb86ef6a20d21e69d4670b895e534e not found: ID does not exist" containerID="65d3dcb6fce444f0f0625af3012ea0259aeb86ef6a20d21e69d4670b895e534e"
Oct 31 03:47:16.132937 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:16.132779    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65d3dcb6fce444f0f0625af3012ea0259aeb86ef6a20d21e69d4670b895e534e"} err="failed to get container status \"65d3dcb6fce444f0f0625af3012ea0259aeb86ef6a20d21e69d4670b895e534e\": rpc error: code = NotFound desc = could not find container \"65d3dcb6fce444f0f0625af3012ea0259aeb86ef6a20d21e69d4670b895e534e\": container with ID starting with 65d3dcb6fce444f0f0625af3012ea0259aeb86ef6a20d21e69d4670b895e534e not found: ID does not exist"
Oct 31 03:47:16.272717 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:16.272630    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn"]
Oct 31 03:47:16.277444 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:16.277418    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bb362142-69c7dbff7b-t4wrn"]
Oct 31 03:47:16.612913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:16.612863    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0cd404b-1899-4e68-a8d1-09d37d8cb5e7" path="/var/lib/kubelet/pods/b0cd404b-1899-4e68-a8d1-09d37d8cb5e7/volumes"
Oct 31 03:47:19.003488 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:47:19.003451    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:47:29.037118 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:29.037084    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"]
Oct 31 03:47:29.037591 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:29.037287    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd" podUID="0aa4dc59-0c78-495d-b13a-69545cb0905f" containerName="istio-proxy" containerID="cri-o://488e160c5559c3625f830e8ca954913a28923782a1ce79b992fadbc995bf8e8a" gracePeriod=30
Oct 31 03:47:34.267312 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.267099    4447 generic.go:356] "Generic (PLEG): container finished" podID="0aa4dc59-0c78-495d-b13a-69545cb0905f" containerID="488e160c5559c3625f830e8ca954913a28923782a1ce79b992fadbc995bf8e8a" exitCode=0
Oct 31 03:47:34.267312 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.267244    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd" event={"ID":"0aa4dc59-0c78-495d-b13a-69545cb0905f","Type":"ContainerDied","Data":"488e160c5559c3625f830e8ca954913a28923782a1ce79b992fadbc995bf8e8a"}
Oct 31 03:47:34.332522 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.332497    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:47:34.463015 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.462983    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-token\" (UniqueName: \"kubernetes.io/projected/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-token\") pod \"0aa4dc59-0c78-495d-b13a-69545cb0905f\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") "
Oct 31 03:47:34.463204 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.463053    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-workload-socket\") pod \"0aa4dc59-0c78-495d-b13a-69545cb0905f\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") "
Oct 31 03:47:34.463204 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.463125    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-envoy\") pod \"0aa4dc59-0c78-495d-b13a-69545cb0905f\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") "
Oct 31 03:47:34.463204 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.463167    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-credential-socket\") pod \"0aa4dc59-0c78-495d-b13a-69545cb0905f\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") "
Oct 31 03:47:34.463204 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.463190    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-data\") pod \"0aa4dc59-0c78-495d-b13a-69545cb0905f\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") "
Oct 31 03:47:34.463373 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.463232    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aa4dc59-0c78-495d-b13a-69545cb0905f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa4dc59-0c78-495d-b13a-69545cb0905f-istiod-ca-cert\") pod \"0aa4dc59-0c78-495d-b13a-69545cb0905f\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") "
Oct 31 03:47:34.463373 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.463356    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-podinfo\") pod \"0aa4dc59-0c78-495d-b13a-69545cb0905f\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") "
Oct 31 03:47:34.463373 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.463353    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-workload-socket" pod "0aa4dc59-0c78-495d-b13a-69545cb0905f" (UID: "0aa4dc59-0c78-495d-b13a-69545cb0905f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:34.463517 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.463401    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aa4dc59-0c78-495d-b13a-69545cb0905f-kube-api-access-dfqmr\" (UniqueName: \"kubernetes.io/projected/0aa4dc59-0c78-495d-b13a-69545cb0905f-kube-api-access-dfqmr\") pod \"0aa4dc59-0c78-495d-b13a-69545cb0905f\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") "
Oct 31 03:47:34.463578 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.463448    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-credential-socket" pod "0aa4dc59-0c78-495d-b13a-69545cb0905f" (UID: "0aa4dc59-0c78-495d-b13a-69545cb0905f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:34.463578 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.463493    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-data" pod "0aa4dc59-0c78-495d-b13a-69545cb0905f" (UID: "0aa4dc59-0c78-495d-b13a-69545cb0905f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:34.463687 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.463671    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-workload-certs\") pod \"0aa4dc59-0c78-495d-b13a-69545cb0905f\" (UID: \"0aa4dc59-0c78-495d-b13a-69545cb0905f\") "
Oct 31 03:47:34.464070 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.463803    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aa4dc59-0c78-495d-b13a-69545cb0905f-istiod-ca-cert" pod "0aa4dc59-0c78-495d-b13a-69545cb0905f" (UID: "0aa4dc59-0c78-495d-b13a-69545cb0905f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:47:34.464070 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.463935    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-workload-certs" pod "0aa4dc59-0c78-495d-b13a-69545cb0905f" (UID: "0aa4dc59-0c78-495d-b13a-69545cb0905f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:34.464773 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.464741    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:47:34.464829 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.464775    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:47:34.464829 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.464791    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:47:34.464829 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.464804    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:47:34.464829 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.464817    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa4dc59-0c78-495d-b13a-69545cb0905f-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:47:34.465729 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.465705    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-podinfo" pod "0aa4dc59-0c78-495d-b13a-69545cb0905f" (UID: "0aa4dc59-0c78-495d-b13a-69545cb0905f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:47:34.465916 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.465863    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa4dc59-0c78-495d-b13a-69545cb0905f-kube-api-access-dfqmr" pod "0aa4dc59-0c78-495d-b13a-69545cb0905f" (UID: "0aa4dc59-0c78-495d-b13a-69545cb0905f"). InnerVolumeSpecName "kube-api-access-dfqmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:34.466054 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.466036    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-envoy" pod "0aa4dc59-0c78-495d-b13a-69545cb0905f" (UID: "0aa4dc59-0c78-495d-b13a-69545cb0905f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:34.466116 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.466036    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-token" pod "0aa4dc59-0c78-495d-b13a-69545cb0905f" (UID: "0aa4dc59-0c78-495d-b13a-69545cb0905f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:34.566644 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.566572    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:47:34.566644 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.566599    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfqmr\" (UniqueName: \"kubernetes.io/projected/0aa4dc59-0c78-495d-b13a-69545cb0905f-kube-api-access-dfqmr\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:47:34.566644 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.566610    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:47:34.566644 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:34.566620    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa4dc59-0c78-495d-b13a-69545cb0905f-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:47:35.277661 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:35.277632    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"
Oct 31 03:47:35.277661 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:35.277652    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd" event={"ID":"0aa4dc59-0c78-495d-b13a-69545cb0905f","Type":"ContainerDied","Data":"e9f841bbceb455441a332662966b9fb204f20739f5017864a75132f0a4d0ba4c"}
Oct 31 03:47:35.278218 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:35.277716    4447 scope.go:122] "RemoveContainer" containerID="488e160c5559c3625f830e8ca954913a28923782a1ce79b992fadbc995bf8e8a"
Oct 31 03:47:35.497466 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:35.497425    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"]
Oct 31 03:47:35.548086 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:35.547995    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-slxfd"]
Oct 31 03:47:36.141471 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:36.141442    4447 scope.go:122] "RemoveContainer" containerID="e7dbbf927833729e6d2139c2e95d80020c4900b22c0851be91fe8b3cf1abd150"
Oct 31 03:47:36.155706 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:36.155685    4447 scope.go:122] "RemoveContainer" containerID="15f977929ea6e7ac9a2e4e040a83b945d255d6b51d2a94ce7ef068997888e1aa"
Oct 31 03:47:36.668388 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:36.667935    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aa4dc59-0c78-495d-b13a-69545cb0905f" path="/var/lib/kubelet/pods/0aa4dc59-0c78-495d-b13a-69545cb0905f/volumes"
Oct 31 03:47:36.833636 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:36.833609    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd"]
Oct 31 03:47:36.836753 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:36.835367    4447 cpu_manager.go:414] "containerMap: removing container" podUID="01a1ebb8-9905-470d-bace-caf42a18af34" containerName="istio-proxy"
Oct 31 03:47:36.836753 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:36.835390    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a1ebb8-9905-470d-bace-caf42a18af34" containerName="istio-proxy"
Oct 31 03:47:36.836753 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:36.835414    4447 cpu_manager.go:414] "containerMap: removing container" podUID="b37925d4-a955-4e16-a0bc-2e9e98ae3d60" containerName="payload-pre-processing"
Oct 31 03:47:36.836753 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:36.835422    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b37925d4-a955-4e16-a0bc-2e9e98ae3d60" containerName="payload-pre-processing"
Oct 31 03:47:36.836753 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:36.835452    4447 cpu_manager.go:414] "containerMap: removing container" podUID="b0cd404b-1899-4e68-a8d1-09d37d8cb5e7" containerName="maas-api"
Oct 31 03:47:36.836753 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:36.835459    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0cd404b-1899-4e68-a8d1-09d37d8cb5e7" containerName="maas-api"
Oct 31 03:47:36.836753 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:36.835478    4447 cpu_manager.go:414] "containerMap: removing container" podUID="0aa4dc59-0c78-495d-b13a-69545cb0905f" containerName="istio-proxy"
Oct 31 03:47:36.836753 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:36.835487    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aa4dc59-0c78-495d-b13a-69545cb0905f" containerName="istio-proxy"
Oct 31 03:47:36.836753 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:36.836025    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a1ebb8-9905-470d-bace-caf42a18af34" containerName="istio-proxy"
Oct 31 03:47:36.836753 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:36.836050    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="b37925d4-a955-4e16-a0bc-2e9e98ae3d60" containerName="payload-pre-processing"
Oct 31 03:47:36.836753 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:36.836067    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0cd404b-1899-4e68-a8d1-09d37d8cb5e7" containerName="maas-api"
Oct 31 03:47:36.836753 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:36.836085    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aa4dc59-0c78-495d-b13a-69545cb0905f" containerName="istio-proxy"
Oct 31 03:47:36.844463 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:36.844435    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd"
Oct 31 03:47:36.897230 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:36.897199    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd"]
Oct 31 03:47:36.923575 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:36.923481    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a2c31a7-238b-40e1-8b37-2aa7e59949c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd\" (UID: \"6a2c31a7-238b-40e1-8b37-2aa7e59949c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd"
Oct 31 03:47:36.923761 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:36.923682    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w8bw\" (UniqueName: \"kubernetes.io/projected/6a2c31a7-238b-40e1-8b37-2aa7e59949c9-kube-api-access-5w8bw\") pod \"payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd\" (UID: \"6a2c31a7-238b-40e1-8b37-2aa7e59949c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd"
Oct 31 03:47:36.946279 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:36.946077    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-2accc9\""
Oct 31 03:47:36.954313 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:36.951779    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-2accc9-dockercfg-6xg7r\""
Oct 31 03:47:37.028954 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:37.028859    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a2c31a7-238b-40e1-8b37-2aa7e59949c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd\" (UID: \"6a2c31a7-238b-40e1-8b37-2aa7e59949c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd"
Oct 31 03:47:37.029152 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:37.029055    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w8bw\" (UniqueName: \"kubernetes.io/projected/6a2c31a7-238b-40e1-8b37-2aa7e59949c9-kube-api-access-5w8bw\") pod \"payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd\" (UID: \"6a2c31a7-238b-40e1-8b37-2aa7e59949c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd"
Oct 31 03:47:37.030304 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:37.030264    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a2c31a7-238b-40e1-8b37-2aa7e59949c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd\" (UID: \"6a2c31a7-238b-40e1-8b37-2aa7e59949c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd"
Oct 31 03:47:37.041389 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:37.040683    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76"]
Oct 31 03:47:37.048563 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:37.048535    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76"
Oct 31 03:47:37.107361 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:37.107319    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76"]
Oct 31 03:47:37.118976 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:37.113576    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w8bw\" (UniqueName: \"kubernetes.io/projected/6a2c31a7-238b-40e1-8b37-2aa7e59949c9-kube-api-access-5w8bw\") pod \"payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd\" (UID: \"6a2c31a7-238b-40e1-8b37-2aa7e59949c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd"
Oct 31 03:47:37.134983 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:37.133026    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0567603-ad2d-47c1-af25-d15f5a02bec2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76\" (UID: \"e0567603-ad2d-47c1-af25-d15f5a02bec2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76"
Oct 31 03:47:37.134983 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:37.133266    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntrct\" (UniqueName: \"kubernetes.io/projected/e0567603-ad2d-47c1-af25-d15f5a02bec2-kube-api-access-ntrct\") pod \"payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76\" (UID: \"e0567603-ad2d-47c1-af25-d15f5a02bec2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76"
Oct 31 03:47:37.161875 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:37.161842    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd"
Oct 31 03:47:37.235124 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:37.235088    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntrct\" (UniqueName: \"kubernetes.io/projected/e0567603-ad2d-47c1-af25-d15f5a02bec2-kube-api-access-ntrct\") pod \"payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76\" (UID: \"e0567603-ad2d-47c1-af25-d15f5a02bec2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76"
Oct 31 03:47:37.235281 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:37.235265    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0567603-ad2d-47c1-af25-d15f5a02bec2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76\" (UID: \"e0567603-ad2d-47c1-af25-d15f5a02bec2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76"
Oct 31 03:47:37.236067 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:37.236020    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0567603-ad2d-47c1-af25-d15f5a02bec2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76\" (UID: \"e0567603-ad2d-47c1-af25-d15f5a02bec2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76"
Oct 31 03:47:37.299290 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:37.299259    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntrct\" (UniqueName: \"kubernetes.io/projected/e0567603-ad2d-47c1-af25-d15f5a02bec2-kube-api-access-ntrct\") pod \"payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76\" (UID: \"e0567603-ad2d-47c1-af25-d15f5a02bec2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76"
Oct 31 03:47:37.414648 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:37.414496    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76"
Oct 31 03:47:37.591168 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:37.591132    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd"]
Oct 31 03:47:37.603331 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:47:37.602316    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a2c31a7_238b_40e1_8b37_2aa7e59949c9.slice/crio-6eeb1448b4f09ef376c7d0871ed6ecbc2bad02291c761f0819bc8020497781b8 WatchSource:0}: Error finding container 6eeb1448b4f09ef376c7d0871ed6ecbc2bad02291c761f0819bc8020497781b8: Status 404 returned error can't find the container with id 6eeb1448b4f09ef376c7d0871ed6ecbc2bad02291c761f0819bc8020497781b8
Oct 31 03:47:37.815517 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:47:37.815482    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0567603_ad2d_47c1_af25_d15f5a02bec2.slice/crio-05a4444e8a9658e37bd8350044abaf02ec73b9ffd02c98a6139295ec3161b1b7 WatchSource:0}: Error finding container 05a4444e8a9658e37bd8350044abaf02ec73b9ffd02c98a6139295ec3161b1b7: Status 404 returned error can't find the container with id 05a4444e8a9658e37bd8350044abaf02ec73b9ffd02c98a6139295ec3161b1b7
Oct 31 03:47:37.816015 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:37.815661    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76"]
Oct 31 03:47:38.306971 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:38.306928    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76" event={"ID":"e0567603-ad2d-47c1-af25-d15f5a02bec2","Type":"ContainerStarted","Data":"7df9d1af4da175890bcec704d80bf808072c944c1f5497acf0bf54b2db8ed1c2"}
Oct 31 03:47:38.307157 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:38.307001    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76"
Oct 31 03:47:38.307157 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:38.307025    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76" event={"ID":"e0567603-ad2d-47c1-af25-d15f5a02bec2","Type":"ContainerStarted","Data":"05a4444e8a9658e37bd8350044abaf02ec73b9ffd02c98a6139295ec3161b1b7"}
Oct 31 03:47:38.308681 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:38.308656    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd" event={"ID":"6a2c31a7-238b-40e1-8b37-2aa7e59949c9","Type":"ContainerStarted","Data":"c9166d57774b9cc459bb62d6544eb207a17624f33972b7a516fdbd396aa5f4b3"}
Oct 31 03:47:38.308809 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:38.308703    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd"
Oct 31 03:47:38.308809 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:38.308718    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd" event={"ID":"6a2c31a7-238b-40e1-8b37-2aa7e59949c9","Type":"ContainerStarted","Data":"6eeb1448b4f09ef376c7d0871ed6ecbc2bad02291c761f0819bc8020497781b8"}
Oct 31 03:47:38.425745 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:38.425686    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76" podStartSLOduration=2.425671563 podStartE2EDuration="2.425671563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:38.424858822 +0000 UTC m=+1683.935841199" watchObservedRunningTime="2026-10-31 03:47:38.425671563 +0000 UTC m=+1683.936653978"
Oct 31 03:47:38.541461 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:38.541237    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd" podStartSLOduration=2.541222704 podStartE2EDuration="2.541222704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:38.540585186 +0000 UTC m=+1684.051567563" watchObservedRunningTime="2026-10-31 03:47:38.541222704 +0000 UTC m=+1684.052205080"
Oct 31 03:47:49.012358 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:47:49.012316    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:47:49.320437 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:49.320354    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd"
Oct 31 03:47:49.320437 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:49.320429    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76"
Oct 31 03:47:57.597846 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:57.597811    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk"]
Oct 31 03:47:57.710996 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:47:57.710962    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f"]
Oct 31 03:48:02.598708 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:02.598664    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk" podUID="95676bc2-e87a-4f52-8740-768b7f2a48c5" containerName="payload-processing" containerID="cri-o://6e6a0542579d6c7ee59f2743319208663a240502ec524c47ef9a543705ed7746" gracePeriod=25
Oct 31 03:48:02.712062 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:02.712016    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f" podUID="1f0836ef-5542-45f6-8385-f71587717e3f" containerName="payload-pre-processing" containerID="cri-o://c6e5b79f31bd003aa0c2ca78f126d946275defc1aad73fdf76d4c8a5f28020a4" gracePeriod=25
Oct 31 03:48:02.897576 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:02.897549    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk"
Oct 31 03:48:03.030066 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.030040    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f"
Oct 31 03:48:03.073914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.073853    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95676bc2-e87a-4f52-8740-768b7f2a48c5-kube-api-access-f8n7j\" (UniqueName: \"kubernetes.io/projected/95676bc2-e87a-4f52-8740-768b7f2a48c5-kube-api-access-f8n7j\") pod \"95676bc2-e87a-4f52-8740-768b7f2a48c5\" (UID: \"95676bc2-e87a-4f52-8740-768b7f2a48c5\") "
Oct 31 03:48:03.074207 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.074185    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95676bc2-e87a-4f52-8740-768b7f2a48c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95676bc2-e87a-4f52-8740-768b7f2a48c5-plugins-config-volume\") pod \"95676bc2-e87a-4f52-8740-768b7f2a48c5\" (UID: \"95676bc2-e87a-4f52-8740-768b7f2a48c5\") "
Oct 31 03:48:03.074615 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.074574    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95676bc2-e87a-4f52-8740-768b7f2a48c5-plugins-config-volume" pod "95676bc2-e87a-4f52-8740-768b7f2a48c5" (UID: "95676bc2-e87a-4f52-8740-768b7f2a48c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:03.074995 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.074973    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95676bc2-e87a-4f52-8740-768b7f2a48c5-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:48:03.076233 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.076209    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95676bc2-e87a-4f52-8740-768b7f2a48c5-kube-api-access-f8n7j" pod "95676bc2-e87a-4f52-8740-768b7f2a48c5" (UID: "95676bc2-e87a-4f52-8740-768b7f2a48c5"). InnerVolumeSpecName "kube-api-access-f8n7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:03.176064 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.175975    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f0836ef-5542-45f6-8385-f71587717e3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f0836ef-5542-45f6-8385-f71587717e3f-plugins-config-volume\") pod \"1f0836ef-5542-45f6-8385-f71587717e3f\" (UID: \"1f0836ef-5542-45f6-8385-f71587717e3f\") "
Oct 31 03:48:03.176218 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.176195    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f0836ef-5542-45f6-8385-f71587717e3f-kube-api-access-hjtx5\" (UniqueName: \"kubernetes.io/projected/1f0836ef-5542-45f6-8385-f71587717e3f-kube-api-access-hjtx5\") pod \"1f0836ef-5542-45f6-8385-f71587717e3f\" (UID: \"1f0836ef-5542-45f6-8385-f71587717e3f\") "
Oct 31 03:48:03.176431 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.176407    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f0836ef-5542-45f6-8385-f71587717e3f-plugins-config-volume" pod "1f0836ef-5542-45f6-8385-f71587717e3f" (UID: "1f0836ef-5542-45f6-8385-f71587717e3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:03.176739 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.176720    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8n7j\" (UniqueName: \"kubernetes.io/projected/95676bc2-e87a-4f52-8740-768b7f2a48c5-kube-api-access-f8n7j\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:48:03.177012 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.176740    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f0836ef-5542-45f6-8385-f71587717e3f-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:48:03.178456 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.178426    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f0836ef-5542-45f6-8385-f71587717e3f-kube-api-access-hjtx5" pod "1f0836ef-5542-45f6-8385-f71587717e3f" (UID: "1f0836ef-5542-45f6-8385-f71587717e3f"). InnerVolumeSpecName "kube-api-access-hjtx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:03.278398 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.278359    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjtx5\" (UniqueName: \"kubernetes.io/projected/1f0836ef-5542-45f6-8385-f71587717e3f-kube-api-access-hjtx5\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:48:03.519258 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.519223    4447 generic.go:356] "Generic (PLEG): container finished" podID="95676bc2-e87a-4f52-8740-768b7f2a48c5" containerID="6e6a0542579d6c7ee59f2743319208663a240502ec524c47ef9a543705ed7746" exitCode=0
Oct 31 03:48:03.519447 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.519266    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk"
Oct 31 03:48:03.519447 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.519309    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk" event={"ID":"95676bc2-e87a-4f52-8740-768b7f2a48c5","Type":"ContainerDied","Data":"6e6a0542579d6c7ee59f2743319208663a240502ec524c47ef9a543705ed7746"}
Oct 31 03:48:03.519447 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.519361    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk" event={"ID":"95676bc2-e87a-4f52-8740-768b7f2a48c5","Type":"ContainerDied","Data":"e0b59575c1ee5c847e6e0d2b03c8ef40b21824b865a4c0474c750d287a016098"}
Oct 31 03:48:03.519447 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.519387    4447 scope.go:122] "RemoveContainer" containerID="6e6a0542579d6c7ee59f2743319208663a240502ec524c47ef9a543705ed7746"
Oct 31 03:48:03.520858 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.520835    4447 generic.go:356] "Generic (PLEG): container finished" podID="1f0836ef-5542-45f6-8385-f71587717e3f" containerID="c6e5b79f31bd003aa0c2ca78f126d946275defc1aad73fdf76d4c8a5f28020a4" exitCode=0
Oct 31 03:48:03.520998 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.520926    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f"
Oct 31 03:48:03.520998 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.520942    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f" event={"ID":"1f0836ef-5542-45f6-8385-f71587717e3f","Type":"ContainerDied","Data":"c6e5b79f31bd003aa0c2ca78f126d946275defc1aad73fdf76d4c8a5f28020a4"}
Oct 31 03:48:03.520998 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.520999    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f" event={"ID":"1f0836ef-5542-45f6-8385-f71587717e3f","Type":"ContainerDied","Data":"cf32e0bceff3cea9b19d7abbf61dd1fea4294bf2024f910295d8624efe289cd9"}
Oct 31 03:48:03.530603 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.530586    4447 scope.go:122] "RemoveContainer" containerID="6e6a0542579d6c7ee59f2743319208663a240502ec524c47ef9a543705ed7746"
Oct 31 03:48:03.530863 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:48:03.530837    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e6a0542579d6c7ee59f2743319208663a240502ec524c47ef9a543705ed7746\": container with ID starting with 6e6a0542579d6c7ee59f2743319208663a240502ec524c47ef9a543705ed7746 not found: ID does not exist" containerID="6e6a0542579d6c7ee59f2743319208663a240502ec524c47ef9a543705ed7746"
Oct 31 03:48:03.530958 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.530877    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e6a0542579d6c7ee59f2743319208663a240502ec524c47ef9a543705ed7746"} err="failed to get container status \"6e6a0542579d6c7ee59f2743319208663a240502ec524c47ef9a543705ed7746\": rpc error: code = NotFound desc = could not find container \"6e6a0542579d6c7ee59f2743319208663a240502ec524c47ef9a543705ed7746\": container with ID starting with 6e6a0542579d6c7ee59f2743319208663a240502ec524c47ef9a543705ed7746 not found: ID does not exist"
Oct 31 03:48:03.530958 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.530947    4447 scope.go:122] "RemoveContainer" containerID="c6e5b79f31bd003aa0c2ca78f126d946275defc1aad73fdf76d4c8a5f28020a4"
Oct 31 03:48:03.545397 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.545377    4447 scope.go:122] "RemoveContainer" containerID="c6e5b79f31bd003aa0c2ca78f126d946275defc1aad73fdf76d4c8a5f28020a4"
Oct 31 03:48:03.545719 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:48:03.545695    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6e5b79f31bd003aa0c2ca78f126d946275defc1aad73fdf76d4c8a5f28020a4\": container with ID starting with c6e5b79f31bd003aa0c2ca78f126d946275defc1aad73fdf76d4c8a5f28020a4 not found: ID does not exist" containerID="c6e5b79f31bd003aa0c2ca78f126d946275defc1aad73fdf76d4c8a5f28020a4"
Oct 31 03:48:03.545784 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.545732    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e5b79f31bd003aa0c2ca78f126d946275defc1aad73fdf76d4c8a5f28020a4"} err="failed to get container status \"c6e5b79f31bd003aa0c2ca78f126d946275defc1aad73fdf76d4c8a5f28020a4\": rpc error: code = NotFound desc = could not find container \"c6e5b79f31bd003aa0c2ca78f126d946275defc1aad73fdf76d4c8a5f28020a4\": container with ID starting with c6e5b79f31bd003aa0c2ca78f126d946275defc1aad73fdf76d4c8a5f28020a4 not found: ID does not exist"
Oct 31 03:48:03.688310 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.688280    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f"]
Oct 31 03:48:03.736803 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.736772    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-71186061-5c69c5dc75-7dc8f"]
Oct 31 03:48:03.801560 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.801471    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk"]
Oct 31 03:48:03.806112 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:03.806090    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-71186061-8684d45bc9-cfsfk"]
Oct 31 03:48:04.611702 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:04.611665    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f0836ef-5542-45f6-8385-f71587717e3f" path="/var/lib/kubelet/pods/1f0836ef-5542-45f6-8385-f71587717e3f/volumes"
Oct 31 03:48:04.612284 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:48:04.612267    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95676bc2-e87a-4f52-8740-768b7f2a48c5" path="/var/lib/kubelet/pods/95676bc2-e87a-4f52-8740-768b7f2a48c5/volumes"
Oct 31 03:48:19.004938 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:48:19.004898    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:48:49.004495 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:48:49.004411    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:49:19.077682 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:49:19.077645    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:49:24.761948 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:24.761919    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"]
Oct 31 03:49:31.766084 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:31.766048    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb"]
Oct 31 03:49:31.888527 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:31.888493    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv"]
Oct 31 03:49:34.904670 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:34.904638    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 03:49:34.922663 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:34.922636    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 03:49:36.766680 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:36.766640    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb" podUID="bec36425-b986-4ea2-9daa-c21328d0e8df" containerName="payload-processing" containerID="cri-o://c52ca1ea78ccdc54023cf97a5500ca3d9a4e9b51ddb13dff18b3b2c7163d48d4" gracePeriod=25
Oct 31 03:49:36.888838 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:36.888800    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv" podUID="e6d58e8e-588d-43e8-808a-d672b68c033e" containerName="payload-pre-processing" containerID="cri-o://2a1cb53180646242eecc7cf34606edb05cff26e1ab82d79c6c2d156ea49dd904" gracePeriod=25
Oct 31 03:49:39.797910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:39.797079    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" podUID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" containerName="main" containerID="cri-o://54a2908d8f0e8c52dc5ffcc83ce6a5d6fd68f5c70d5cadf0526bb955457dcbd9" gracePeriod=45
Oct 31 03:49:40.140722 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.140691    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:49:40.206298 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.206266    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-kube-api-access-g2wwq\" (UniqueName: \"kubernetes.io/projected/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-kube-api-access-g2wwq\") pod \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\" (UID: \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\") "
Oct 31 03:49:40.206489 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.206419    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-dshm\") pod \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\" (UID: \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\") "
Oct 31 03:49:40.206489 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.206480    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-tls-certs\") pod \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\" (UID: \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\") "
Oct 31 03:49:40.206625 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.206611    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-model-cache\") pod \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\" (UID: \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\") "
Oct 31 03:49:40.206698 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.206686    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-tmp-dir\") pod \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\" (UID: \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\") "
Oct 31 03:49:40.206780 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.206767    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-home\" (UniqueName: \"kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-home\") pod \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\" (UID: \"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6\") "
Oct 31 03:49:40.207014 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.206949    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-model-cache" pod "c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" (UID: "c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:49:40.207373 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.207301    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-home" pod "c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" (UID: "c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:49:40.207485 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.206962    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-tmp-dir" pod "c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" (UID: "c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:49:40.208108 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.208082    4447 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-model-cache\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:49:40.208216 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.208110    4447 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-tmp-dir\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:49:40.208216 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.208125    4447 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-home\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:49:40.208865 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.208839    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-tls-certs" pod "c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" (UID: "c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:49:40.208991 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.208928    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-dshm" pod "c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" (UID: "c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:49:40.208991 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.208968    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-kube-api-access-g2wwq" pod "c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" (UID: "c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6"). InnerVolumeSpecName "kube-api-access-g2wwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:49:40.309512 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.309474    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2wwq\" (UniqueName: \"kubernetes.io/projected/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-kube-api-access-g2wwq\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:49:40.309512 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.309508    4447 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-dshm\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:49:40.309512 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.309520    4447 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6-tls-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:49:40.435077 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.434994    4447 generic.go:356] "Generic (PLEG): container finished" podID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" containerID="54a2908d8f0e8c52dc5ffcc83ce6a5d6fd68f5c70d5cadf0526bb955457dcbd9" exitCode=0
Oct 31 03:49:40.435077 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.435056    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"
Oct 31 03:49:40.435263 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.435078    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" event={"ID":"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6","Type":"ContainerDied","Data":"54a2908d8f0e8c52dc5ffcc83ce6a5d6fd68f5c70d5cadf0526bb955457dcbd9"}
Oct 31 03:49:40.435263 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.435122    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f" event={"ID":"c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6","Type":"ContainerDied","Data":"578bbbc6b884590cadb8e8bacde128b1cdebb5b88a52f505042d5eac30b9ac06"}
Oct 31 03:49:40.435263 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.435139    4447 scope.go:122] "RemoveContainer" containerID="54a2908d8f0e8c52dc5ffcc83ce6a5d6fd68f5c70d5cadf0526bb955457dcbd9"
Oct 31 03:49:40.447134 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.447110    4447 scope.go:122] "RemoveContainer" containerID="54a2908d8f0e8c52dc5ffcc83ce6a5d6fd68f5c70d5cadf0526bb955457dcbd9"
Oct 31 03:49:40.447459 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:49:40.447439    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54a2908d8f0e8c52dc5ffcc83ce6a5d6fd68f5c70d5cadf0526bb955457dcbd9\": container with ID starting with 54a2908d8f0e8c52dc5ffcc83ce6a5d6fd68f5c70d5cadf0526bb955457dcbd9 not found: ID does not exist" containerID="54a2908d8f0e8c52dc5ffcc83ce6a5d6fd68f5c70d5cadf0526bb955457dcbd9"
Oct 31 03:49:40.447516 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.447469    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a2908d8f0e8c52dc5ffcc83ce6a5d6fd68f5c70d5cadf0526bb955457dcbd9"} err="failed to get container status \"54a2908d8f0e8c52dc5ffcc83ce6a5d6fd68f5c70d5cadf0526bb955457dcbd9\": rpc error: code = NotFound desc = could not find container \"54a2908d8f0e8c52dc5ffcc83ce6a5d6fd68f5c70d5cadf0526bb955457dcbd9\": container with ID starting with 54a2908d8f0e8c52dc5ffcc83ce6a5d6fd68f5c70d5cadf0526bb955457dcbd9 not found: ID does not exist"
Oct 31 03:49:40.611903 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.611851    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"]
Oct 31 03:49:40.616721 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:40.616697    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-096812/simulator-w4-096812-kserve-5d865c545-4hk7f"]
Oct 31 03:49:42.610768 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:42.610732    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" path="/var/lib/kubelet/pods/c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6/volumes"
Oct 31 03:49:45.460067 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:45.460027    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb" podUID="bec36425-b986-4ea2-9daa-c21328d0e8df" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:49:45.697811 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:45.697777    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"]
Oct 31 03:49:45.698064 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:45.698040    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn" podUID="9f503715-aa26-4a05-9391-de0b40a919e7" containerName="istio-proxy" containerID="cri-o://f0b290178f5a30e4f2878a124b1baae346018dac8287fcd5c025dd04dd6ff3f0" gracePeriod=30
Oct 31 03:49:46.466873 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:46.466834    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv" podUID="e6d58e8e-588d-43e8-808a-d672b68c033e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:49:49.004074 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:49:49.004035    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:49:50.998526 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:50.998498    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:49:51.107320 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.107287    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f503715-aa26-4a05-9391-de0b40a919e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f503715-aa26-4a05-9391-de0b40a919e7-istiod-ca-cert\") pod \"9f503715-aa26-4a05-9391-de0b40a919e7\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") "
Oct 31 03:49:51.107487 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.107341    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f503715-aa26-4a05-9391-de0b40a919e7-istio-token\" (UniqueName: \"kubernetes.io/projected/9f503715-aa26-4a05-9391-de0b40a919e7-istio-token\") pod \"9f503715-aa26-4a05-9391-de0b40a919e7\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") "
Oct 31 03:49:51.107487 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.107363    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-istio-envoy\") pod \"9f503715-aa26-4a05-9391-de0b40a919e7\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") "
Oct 31 03:49:51.107487 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.107392    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-istio-data\") pod \"9f503715-aa26-4a05-9391-de0b40a919e7\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") "
Oct 31 03:49:51.107585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.107505    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-workload-certs\") pod \"9f503715-aa26-4a05-9391-de0b40a919e7\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") "
Oct 31 03:49:51.107585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.107573    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-credential-socket\") pod \"9f503715-aa26-4a05-9391-de0b40a919e7\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") "
Oct 31 03:49:51.107662 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.107650    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f503715-aa26-4a05-9391-de0b40a919e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f503715-aa26-4a05-9391-de0b40a919e7-istio-podinfo\") pod \"9f503715-aa26-4a05-9391-de0b40a919e7\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") "
Oct 31 03:49:51.107713 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.107702    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f503715-aa26-4a05-9391-de0b40a919e7-kube-api-access-82jvr\" (UniqueName: \"kubernetes.io/projected/9f503715-aa26-4a05-9391-de0b40a919e7-kube-api-access-82jvr\") pod \"9f503715-aa26-4a05-9391-de0b40a919e7\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") "
Oct 31 03:49:51.107767 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.107747    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-workload-socket\") pod \"9f503715-aa26-4a05-9391-de0b40a919e7\" (UID: \"9f503715-aa26-4a05-9391-de0b40a919e7\") "
Oct 31 03:49:51.107819 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.107761    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-istio-data" pod "9f503715-aa26-4a05-9391-de0b40a919e7" (UID: "9f503715-aa26-4a05-9391-de0b40a919e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:49:51.107819 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.107794    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f503715-aa26-4a05-9391-de0b40a919e7-istiod-ca-cert" pod "9f503715-aa26-4a05-9391-de0b40a919e7" (UID: "9f503715-aa26-4a05-9391-de0b40a919e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:49:51.107965 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.107919    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-workload-certs" pod "9f503715-aa26-4a05-9391-de0b40a919e7" (UID: "9f503715-aa26-4a05-9391-de0b40a919e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:49:51.108020 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.107997    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-credential-socket" pod "9f503715-aa26-4a05-9391-de0b40a919e7" (UID: "9f503715-aa26-4a05-9391-de0b40a919e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:49:51.108374 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.108340    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-workload-socket" pod "9f503715-aa26-4a05-9391-de0b40a919e7" (UID: "9f503715-aa26-4a05-9391-de0b40a919e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:49:51.108576 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.108555    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:49:51.108658 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.108584    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:49:51.108658 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.108602    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:49:51.108658 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.108616    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f503715-aa26-4a05-9391-de0b40a919e7-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:49:51.108658 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.108631    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:49:51.110362 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.110334    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f503715-aa26-4a05-9391-de0b40a919e7-istio-token" pod "9f503715-aa26-4a05-9391-de0b40a919e7" (UID: "9f503715-aa26-4a05-9391-de0b40a919e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:49:51.110471 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.110420    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-istio-envoy" pod "9f503715-aa26-4a05-9391-de0b40a919e7" (UID: "9f503715-aa26-4a05-9391-de0b40a919e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:49:51.110653 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.110629    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f503715-aa26-4a05-9391-de0b40a919e7-kube-api-access-82jvr" pod "9f503715-aa26-4a05-9391-de0b40a919e7" (UID: "9f503715-aa26-4a05-9391-de0b40a919e7"). InnerVolumeSpecName "kube-api-access-82jvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:49:51.111185 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.111166    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f503715-aa26-4a05-9391-de0b40a919e7-istio-podinfo" pod "9f503715-aa26-4a05-9391-de0b40a919e7" (UID: "9f503715-aa26-4a05-9391-de0b40a919e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:49:51.210585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.210540    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f503715-aa26-4a05-9391-de0b40a919e7-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:49:51.210585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.210585    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82jvr\" (UniqueName: \"kubernetes.io/projected/9f503715-aa26-4a05-9391-de0b40a919e7-kube-api-access-82jvr\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:49:51.210792 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.210602    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f503715-aa26-4a05-9391-de0b40a919e7-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:49:51.210792 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.210616    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f503715-aa26-4a05-9391-de0b40a919e7-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:49:51.527872 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.527840    4447 generic.go:356] "Generic (PLEG): container finished" podID="9f503715-aa26-4a05-9391-de0b40a919e7" containerID="f0b290178f5a30e4f2878a124b1baae346018dac8287fcd5c025dd04dd6ff3f0" exitCode=0
Oct 31 03:49:51.528074 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.527904    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn" event={"ID":"9f503715-aa26-4a05-9391-de0b40a919e7","Type":"ContainerDied","Data":"f0b290178f5a30e4f2878a124b1baae346018dac8287fcd5c025dd04dd6ff3f0"}
Oct 31 03:49:51.528074 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.527916    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"
Oct 31 03:49:51.528074 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.527967    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn" event={"ID":"9f503715-aa26-4a05-9391-de0b40a919e7","Type":"ContainerDied","Data":"fa3572d27a6b219eb7f51422ff2ca99de8cbb4fe31f8c43902ce180d3a3748a5"}
Oct 31 03:49:51.528074 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.527989    4447 scope.go:122] "RemoveContainer" containerID="f0b290178f5a30e4f2878a124b1baae346018dac8287fcd5c025dd04dd6ff3f0"
Oct 31 03:49:51.540377 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.540357    4447 scope.go:122] "RemoveContainer" containerID="f0b290178f5a30e4f2878a124b1baae346018dac8287fcd5c025dd04dd6ff3f0"
Oct 31 03:49:51.540697 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:49:51.540666    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0b290178f5a30e4f2878a124b1baae346018dac8287fcd5c025dd04dd6ff3f0\": container with ID starting with f0b290178f5a30e4f2878a124b1baae346018dac8287fcd5c025dd04dd6ff3f0 not found: ID does not exist" containerID="f0b290178f5a30e4f2878a124b1baae346018dac8287fcd5c025dd04dd6ff3f0"
Oct 31 03:49:51.540775 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.540708    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0b290178f5a30e4f2878a124b1baae346018dac8287fcd5c025dd04dd6ff3f0"} err="failed to get container status \"f0b290178f5a30e4f2878a124b1baae346018dac8287fcd5c025dd04dd6ff3f0\": rpc error: code = NotFound desc = could not find container \"f0b290178f5a30e4f2878a124b1baae346018dac8287fcd5c025dd04dd6ff3f0\": container with ID starting with f0b290178f5a30e4f2878a124b1baae346018dac8287fcd5c025dd04dd6ff3f0 not found: ID does not exist"
Oct 31 03:49:51.709977 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.709941    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"]
Oct 31 03:49:51.714651 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:51.714624    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-6znhn"]
Oct 31 03:49:52.736296 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:52.736263    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f503715-aa26-4a05-9391-de0b40a919e7" path="/var/lib/kubelet/pods/9f503715-aa26-4a05-9391-de0b40a919e7/volumes"
Oct 31 03:49:55.460295 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:55.460258    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb" podUID="bec36425-b986-4ea2-9daa-c21328d0e8df" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:49:56.466739 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:56.466706    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv" podUID="e6d58e8e-588d-43e8-808a-d672b68c033e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:49:57.251419 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.251388    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb"
Oct 31 03:49:57.261985 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.261949    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bec36425-b986-4ea2-9daa-c21328d0e8df-kube-api-access-h9mpk\" (UniqueName: \"kubernetes.io/projected/bec36425-b986-4ea2-9daa-c21328d0e8df-kube-api-access-h9mpk\") pod \"bec36425-b986-4ea2-9daa-c21328d0e8df\" (UID: \"bec36425-b986-4ea2-9daa-c21328d0e8df\") "
Oct 31 03:49:57.262227 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.262207    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bec36425-b986-4ea2-9daa-c21328d0e8df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bec36425-b986-4ea2-9daa-c21328d0e8df-plugins-config-volume\") pod \"bec36425-b986-4ea2-9daa-c21328d0e8df\" (UID: \"bec36425-b986-4ea2-9daa-c21328d0e8df\") "
Oct 31 03:49:57.262713 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.262651    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bec36425-b986-4ea2-9daa-c21328d0e8df-plugins-config-volume" pod "bec36425-b986-4ea2-9daa-c21328d0e8df" (UID: "bec36425-b986-4ea2-9daa-c21328d0e8df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:49:57.263088 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.263068    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bec36425-b986-4ea2-9daa-c21328d0e8df-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:49:57.264619 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.264592    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec36425-b986-4ea2-9daa-c21328d0e8df-kube-api-access-h9mpk" pod "bec36425-b986-4ea2-9daa-c21328d0e8df" (UID: "bec36425-b986-4ea2-9daa-c21328d0e8df"). InnerVolumeSpecName "kube-api-access-h9mpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:49:57.365438 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.365402    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9mpk\" (UniqueName: \"kubernetes.io/projected/bec36425-b986-4ea2-9daa-c21328d0e8df-kube-api-access-h9mpk\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:49:57.387984 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.387958    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv"
Oct 31 03:49:57.467040 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.467012    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6d58e8e-588d-43e8-808a-d672b68c033e-kube-api-access-qs6px\" (UniqueName: \"kubernetes.io/projected/e6d58e8e-588d-43e8-808a-d672b68c033e-kube-api-access-qs6px\") pod \"e6d58e8e-588d-43e8-808a-d672b68c033e\" (UID: \"e6d58e8e-588d-43e8-808a-d672b68c033e\") "
Oct 31 03:49:57.467525 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.467066    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6d58e8e-588d-43e8-808a-d672b68c033e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6d58e8e-588d-43e8-808a-d672b68c033e-plugins-config-volume\") pod \"e6d58e8e-588d-43e8-808a-d672b68c033e\" (UID: \"e6d58e8e-588d-43e8-808a-d672b68c033e\") "
Oct 31 03:49:57.467525 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.467469    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6d58e8e-588d-43e8-808a-d672b68c033e-plugins-config-volume" pod "e6d58e8e-588d-43e8-808a-d672b68c033e" (UID: "e6d58e8e-588d-43e8-808a-d672b68c033e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:49:57.467721 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.467701    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6d58e8e-588d-43e8-808a-d672b68c033e-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:49:57.469406 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.469384    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d58e8e-588d-43e8-808a-d672b68c033e-kube-api-access-qs6px" pod "e6d58e8e-588d-43e8-808a-d672b68c033e" (UID: "e6d58e8e-588d-43e8-808a-d672b68c033e"). InnerVolumeSpecName "kube-api-access-qs6px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:49:57.569096 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.569065    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs6px\" (UniqueName: \"kubernetes.io/projected/e6d58e8e-588d-43e8-808a-d672b68c033e-kube-api-access-qs6px\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:49:57.755980 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.755866    4447 generic.go:356] "Generic (PLEG): container finished" podID="bec36425-b986-4ea2-9daa-c21328d0e8df" containerID="c52ca1ea78ccdc54023cf97a5500ca3d9a4e9b51ddb13dff18b3b2c7163d48d4" exitCode=0
Oct 31 03:49:57.755980 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.755948    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb"
Oct 31 03:49:57.755980 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.755953    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb" event={"ID":"bec36425-b986-4ea2-9daa-c21328d0e8df","Type":"ContainerDied","Data":"c52ca1ea78ccdc54023cf97a5500ca3d9a4e9b51ddb13dff18b3b2c7163d48d4"}
Oct 31 03:49:57.756233 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.756001    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb" event={"ID":"bec36425-b986-4ea2-9daa-c21328d0e8df","Type":"ContainerDied","Data":"d35c5953df8ede80c7cb2087094f7feb1268d05001e8c28dc2c042ae85bd5697"}
Oct 31 03:49:57.756233 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.756022    4447 scope.go:122] "RemoveContainer" containerID="c52ca1ea78ccdc54023cf97a5500ca3d9a4e9b51ddb13dff18b3b2c7163d48d4"
Oct 31 03:49:57.757515 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.757492    4447 generic.go:356] "Generic (PLEG): container finished" podID="e6d58e8e-588d-43e8-808a-d672b68c033e" containerID="2a1cb53180646242eecc7cf34606edb05cff26e1ab82d79c6c2d156ea49dd904" exitCode=0
Oct 31 03:49:57.757624 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.757530    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv"
Oct 31 03:49:57.757624 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.757600    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv" event={"ID":"e6d58e8e-588d-43e8-808a-d672b68c033e","Type":"ContainerDied","Data":"2a1cb53180646242eecc7cf34606edb05cff26e1ab82d79c6c2d156ea49dd904"}
Oct 31 03:49:57.757705 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.757661    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv" event={"ID":"e6d58e8e-588d-43e8-808a-d672b68c033e","Type":"ContainerDied","Data":"c3650c382f1af2f3e31fbca20bb85b4e7e27319ea665b036d9fafebdb9d28805"}
Oct 31 03:49:57.771384 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.771366    4447 scope.go:122] "RemoveContainer" containerID="c52ca1ea78ccdc54023cf97a5500ca3d9a4e9b51ddb13dff18b3b2c7163d48d4"
Oct 31 03:49:57.771698 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:49:57.771677    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c52ca1ea78ccdc54023cf97a5500ca3d9a4e9b51ddb13dff18b3b2c7163d48d4\": container with ID starting with c52ca1ea78ccdc54023cf97a5500ca3d9a4e9b51ddb13dff18b3b2c7163d48d4 not found: ID does not exist" containerID="c52ca1ea78ccdc54023cf97a5500ca3d9a4e9b51ddb13dff18b3b2c7163d48d4"
Oct 31 03:49:57.771749 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.771712    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c52ca1ea78ccdc54023cf97a5500ca3d9a4e9b51ddb13dff18b3b2c7163d48d4"} err="failed to get container status \"c52ca1ea78ccdc54023cf97a5500ca3d9a4e9b51ddb13dff18b3b2c7163d48d4\": rpc error: code = NotFound desc = could not find container \"c52ca1ea78ccdc54023cf97a5500ca3d9a4e9b51ddb13dff18b3b2c7163d48d4\": container with ID starting with c52ca1ea78ccdc54023cf97a5500ca3d9a4e9b51ddb13dff18b3b2c7163d48d4 not found: ID does not exist"
Oct 31 03:49:57.771749 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.771735    4447 scope.go:122] "RemoveContainer" containerID="2a1cb53180646242eecc7cf34606edb05cff26e1ab82d79c6c2d156ea49dd904"
Oct 31 03:49:57.798739 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.798717    4447 scope.go:122] "RemoveContainer" containerID="2a1cb53180646242eecc7cf34606edb05cff26e1ab82d79c6c2d156ea49dd904"
Oct 31 03:49:57.799201 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:49:57.799164    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a1cb53180646242eecc7cf34606edb05cff26e1ab82d79c6c2d156ea49dd904\": container with ID starting with 2a1cb53180646242eecc7cf34606edb05cff26e1ab82d79c6c2d156ea49dd904 not found: ID does not exist" containerID="2a1cb53180646242eecc7cf34606edb05cff26e1ab82d79c6c2d156ea49dd904"
Oct 31 03:49:57.799308 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.799197    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a1cb53180646242eecc7cf34606edb05cff26e1ab82d79c6c2d156ea49dd904"} err="failed to get container status \"2a1cb53180646242eecc7cf34606edb05cff26e1ab82d79c6c2d156ea49dd904\": rpc error: code = NotFound desc = could not find container \"2a1cb53180646242eecc7cf34606edb05cff26e1ab82d79c6c2d156ea49dd904\": container with ID starting with 2a1cb53180646242eecc7cf34606edb05cff26e1ab82d79c6c2d156ea49dd904 not found: ID does not exist"
Oct 31 03:49:57.934954 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.934873    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv"]
Oct 31 03:49:57.937064 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:57.937027    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-096812-9dbb4dbd7-tdggv"]
Oct 31 03:49:58.043307 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:58.043218    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb"]
Oct 31 03:49:58.050788 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:58.049912    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-096812-6f47b8cd6c-wzvrb"]
Oct 31 03:49:58.612159 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:58.612124    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bec36425-b986-4ea2-9daa-c21328d0e8df" path="/var/lib/kubelet/pods/bec36425-b986-4ea2-9daa-c21328d0e8df/volumes"
Oct 31 03:49:58.612671 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:58.612655    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6d58e8e-588d-43e8-808a-d672b68c033e" path="/var/lib/kubelet/pods/e6d58e8e-588d-43e8-808a-d672b68c033e/volumes"
Oct 31 03:49:59.519975 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:59.519940    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"]
Oct 31 03:49:59.520195 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:59.520168    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs" podUID="377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f" containerName="istio-proxy" containerID="cri-o://bb0be6b368ff75326103c6413b3f785eeafcc65f4730fe6e5793d46baa2961a2" gracePeriod=30
Oct 31 03:49:59.824855 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:59.824772    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"]
Oct 31 03:49:59.825281 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:49:59.824981    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt" podUID="29f09a6c-204d-410b-b765-65368fd42c9d" containerName="istio-proxy" containerID="cri-o://126c377ab7e035e92301e97bf83d38983e2d93d1ace3e133925ce0a5852dbc44" gracePeriod=30
Oct 31 03:50:00.165511 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:00.164749    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"]
Oct 31 03:50:00.165511 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:00.165179    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz" podUID="d379c75c-04e8-4ab1-8b40-e8de8b0631ef" containerName="istio-proxy" containerID="cri-o://b698f8298b01106fb48ff2ff22b84a4a816f68fb1099fee487f1c1bb14730390" gracePeriod=30
Oct 31 03:50:00.170337 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:00.170309    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"]
Oct 31 03:50:00.170516 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:00.170495    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd" podUID="24d2a84e-6aad-4669-a456-ebe20cf2f0e9" containerName="istio-proxy" containerID="cri-o://edc9f8388d56defcc6825e96a1fdd571b057cfbf9ebe3aecf80a184b2532f7ee" gracePeriod=30
Oct 31 03:50:00.176938 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:00.176847    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"]
Oct 31 03:50:00.177128 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:00.177100    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5" podUID="f1e5b912-f46c-4ce3-ad7e-521e506e5e88" containerName="istio-proxy" containerID="cri-o://0d37ad73241bf95ccbc852fe988a0f6bc29dfd5e747bf6ac8faf70d99b6077d2" gracePeriod=30
Oct 31 03:50:00.755525 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:00.755488    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"]
Oct 31 03:50:00.755749 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:00.755723    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m" podUID="8954a5a4-2ec5-4443-adef-155c6781cb89" containerName="istio-proxy" containerID="cri-o://4846319f4401a761790dee40aa327c47d9374d74ee3f16042483df9ef8fe3760" gracePeriod=30
Oct 31 03:50:00.827678 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:00.827640    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"]
Oct 31 03:50:00.828773 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:00.828736    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q" podUID="0926be7e-f12d-474d-a3c8-1a3eb3fbc43f" containerName="istio-proxy" containerID="cri-o://aa90022d05a77afcadb5f852f4aa8d24d57b78805cfeb9c03ac6cd7d3bfe2b50" gracePeriod=30
Oct 31 03:50:01.348169 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.348131    4447 patch_prober.go:29] interesting pod/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:50:01.348343 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.348185    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd" podUID="24d2a84e-6aad-4669-a456-ebe20cf2f0e9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:50:01.527623 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.527585    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx"]
Oct 31 03:50:01.529174 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.529146    4447 cpu_manager.go:414] "containerMap: removing container" podUID="95676bc2-e87a-4f52-8740-768b7f2a48c5" containerName="payload-processing"
Oct 31 03:50:01.529174 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.529173    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95676bc2-e87a-4f52-8740-768b7f2a48c5" containerName="payload-processing"
Oct 31 03:50:01.529346 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.529199    4447 cpu_manager.go:414] "containerMap: removing container" podUID="1f0836ef-5542-45f6-8385-f71587717e3f" containerName="payload-pre-processing"
Oct 31 03:50:01.529346 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.529208    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f0836ef-5542-45f6-8385-f71587717e3f" containerName="payload-pre-processing"
Oct 31 03:50:01.529346 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.529234    4447 cpu_manager.go:414] "containerMap: removing container" podUID="9f503715-aa26-4a05-9391-de0b40a919e7" containerName="istio-proxy"
Oct 31 03:50:01.529346 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.529242    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f503715-aa26-4a05-9391-de0b40a919e7" containerName="istio-proxy"
Oct 31 03:50:01.529346 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.529285    4447 cpu_manager.go:414] "containerMap: removing container" podUID="e6d58e8e-588d-43e8-808a-d672b68c033e" containerName="payload-pre-processing"
Oct 31 03:50:01.529346 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.529292    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6d58e8e-588d-43e8-808a-d672b68c033e" containerName="payload-pre-processing"
Oct 31 03:50:01.529346 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.529301    4447 cpu_manager.go:414] "containerMap: removing container" podUID="bec36425-b986-4ea2-9daa-c21328d0e8df" containerName="payload-processing"
Oct 31 03:50:01.529346 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.529308    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bec36425-b986-4ea2-9daa-c21328d0e8df" containerName="payload-processing"
Oct 31 03:50:01.529346 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.529333    4447 cpu_manager.go:414] "containerMap: removing container" podUID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" containerName="main"
Oct 31 03:50:01.529346 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.529339    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" containerName="main"
Oct 31 03:50:01.529673 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.529507    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6d58e8e-588d-43e8-808a-d672b68c033e" containerName="payload-pre-processing"
Oct 31 03:50:01.529673 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.529522    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="bec36425-b986-4ea2-9daa-c21328d0e8df" containerName="payload-processing"
Oct 31 03:50:01.529673 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.529534    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c77031-ee5b-4d25-9b0d-dc7a49aaa1a6" containerName="main"
Oct 31 03:50:01.529673 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.529552    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="95676bc2-e87a-4f52-8740-768b7f2a48c5" containerName="payload-processing"
Oct 31 03:50:01.529673 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.529564    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f0836ef-5542-45f6-8385-f71587717e3f" containerName="payload-pre-processing"
Oct 31 03:50:01.529673 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.529577    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f503715-aa26-4a05-9391-de0b40a919e7" containerName="istio-proxy"
Oct 31 03:50:01.533731 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.533705    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx"
Oct 31 03:50:01.586783 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.586747    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-1783b3-dockercfg-5xm6v\""
Oct 31 03:50:01.589138 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.589089    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-1783b3\""
Oct 31 03:50:01.593018 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.592990    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx"]
Oct 31 03:50:01.641875 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.641793    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh9hz\" (UniqueName: \"kubernetes.io/projected/f334da75-10b1-4320-9d31-4b8d428040d6-kube-api-access-qh9hz\") pod \"payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx\" (UID: \"f334da75-10b1-4320-9d31-4b8d428040d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx"
Oct 31 03:50:01.642197 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.642171    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f334da75-10b1-4320-9d31-4b8d428040d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx\" (UID: \"f334da75-10b1-4320-9d31-4b8d428040d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx"
Oct 31 03:50:01.731707 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.705260    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb"]
Oct 31 03:50:01.733946 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.733859    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb"
Oct 31 03:50:01.740761 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.740725    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb"]
Oct 31 03:50:01.744066 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.744027    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh9hz\" (UniqueName: \"kubernetes.io/projected/f334da75-10b1-4320-9d31-4b8d428040d6-kube-api-access-qh9hz\") pod \"payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx\" (UID: \"f334da75-10b1-4320-9d31-4b8d428040d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx"
Oct 31 03:50:01.744437 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.744275    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f334da75-10b1-4320-9d31-4b8d428040d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx\" (UID: \"f334da75-10b1-4320-9d31-4b8d428040d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx"
Oct 31 03:50:01.745323 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.745256    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f334da75-10b1-4320-9d31-4b8d428040d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx\" (UID: \"f334da75-10b1-4320-9d31-4b8d428040d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx"
Oct 31 03:50:01.831255 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.820951    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7153-bf2e87"]
Oct 31 03:50:01.831255 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.825803    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh9hz\" (UniqueName: \"kubernetes.io/projected/f334da75-10b1-4320-9d31-4b8d428040d6-kube-api-access-qh9hz\") pod \"payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx\" (UID: \"f334da75-10b1-4320-9d31-4b8d428040d6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx"
Oct 31 03:50:01.831255 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.828115    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7153-bf2e87"
Oct 31 03:50:01.850707 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.846110    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrgc8\" (UniqueName: \"kubernetes.io/projected/76378008-52e0-44da-9bf4-7c6745178755-kube-api-access-qrgc8\") pod \"payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb\" (UID: \"76378008-52e0-44da-9bf4-7c6745178755\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb"
Oct 31 03:50:01.862951 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.862363    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx"
Oct 31 03:50:01.863928 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.863230    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76378008-52e0-44da-9bf4-7c6745178755-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb\" (UID: \"76378008-52e0-44da-9bf4-7c6745178755\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb"
Oct 31 03:50:01.889523 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.889279    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hppm7\""
Oct 31 03:50:01.910249 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.910218    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7153-bf2e87"]
Oct 31 03:50:01.970767 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.970618    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76378008-52e0-44da-9bf4-7c6745178755-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb\" (UID: \"76378008-52e0-44da-9bf4-7c6745178755\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb"
Oct 31 03:50:01.971029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.970825    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrgc8\" (UniqueName: \"kubernetes.io/projected/76378008-52e0-44da-9bf4-7c6745178755-kube-api-access-qrgc8\") pod \"payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb\" (UID: \"76378008-52e0-44da-9bf4-7c6745178755\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb"
Oct 31 03:50:01.971111 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.971008    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5j5v\" (UniqueName: \"kubernetes.io/projected/39884c94-9c25-4e4f-b2ff-f4378fb16a7b-kube-api-access-l5j5v\") pod \"test-curl-7153-bf2e87\" (UID: \"39884c94-9c25-4e4f-b2ff-f4378fb16a7b\") " pod="opendatahub/test-curl-7153-bf2e87"
Oct 31 03:50:01.977157 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:01.972266    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76378008-52e0-44da-9bf4-7c6745178755-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb\" (UID: \"76378008-52e0-44da-9bf4-7c6745178755\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb"
Oct 31 03:50:02.035282 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:02.035253    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrgc8\" (UniqueName: \"kubernetes.io/projected/76378008-52e0-44da-9bf4-7c6745178755-kube-api-access-qrgc8\") pod \"payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb\" (UID: \"76378008-52e0-44da-9bf4-7c6745178755\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb"
Oct 31 03:50:02.061250 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:02.061100    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb"
Oct 31 03:50:02.073117 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:02.073089    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5j5v\" (UniqueName: \"kubernetes.io/projected/39884c94-9c25-4e4f-b2ff-f4378fb16a7b-kube-api-access-l5j5v\") pod \"test-curl-7153-bf2e87\" (UID: \"39884c94-9c25-4e4f-b2ff-f4378fb16a7b\") " pod="opendatahub/test-curl-7153-bf2e87"
Oct 31 03:50:02.097594 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:02.097552    4447 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:50:02.097768 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:02.097625    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m" podUID="8954a5a4-2ec5-4443-adef-155c6781cb89" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:50:02.136089 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:02.136061    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5j5v\" (UniqueName: \"kubernetes.io/projected/39884c94-9c25-4e4f-b2ff-f4378fb16a7b-kube-api-access-l5j5v\") pod \"test-curl-7153-bf2e87\" (UID: \"39884c94-9c25-4e4f-b2ff-f4378fb16a7b\") " pod="opendatahub/test-curl-7153-bf2e87"
Oct 31 03:50:02.155480 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:02.155351    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:50:02.155480 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:02.155416    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q" podUID="0926be7e-f12d-474d-a3c8-1a3eb3fbc43f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:50:02.242049 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:02.242002    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7153-bf2e87"
Oct 31 03:50:02.282551 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:50:02.282521    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf334da75_10b1_4320_9d31_4b8d428040d6.slice/crio-4ed6d2c566b23a6e33cdd39f89d0c4f62a8a745bce4e4eb81e95e35c20cda8dd WatchSource:0}: Error finding container 4ed6d2c566b23a6e33cdd39f89d0c4f62a8a745bce4e4eb81e95e35c20cda8dd: Status 404 returned error can't find the container with id 4ed6d2c566b23a6e33cdd39f89d0c4f62a8a745bce4e4eb81e95e35c20cda8dd
Oct 31 03:50:02.283422 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:02.283391    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx"]
Oct 31 03:50:02.308569 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:02.308536    4447 patch_prober.go:29] interesting pod/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:50:02.308712 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:02.308594    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt" podUID="29f09a6c-204d-410b-b765-65368fd42c9d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:50:02.448124 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:50:02.448091    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76378008_52e0_44da_9bf4_7c6745178755.slice/crio-b8d44e98fbaeb72fb89bdbf8e48cd68377d7288ecb9a5e41b2f799b6febb3a21 WatchSource:0}: Error finding container b8d44e98fbaeb72fb89bdbf8e48cd68377d7288ecb9a5e41b2f799b6febb3a21: Status 404 returned error can't find the container with id b8d44e98fbaeb72fb89bdbf8e48cd68377d7288ecb9a5e41b2f799b6febb3a21
Oct 31 03:50:02.448356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:02.448333    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb"]
Oct 31 03:50:02.624081 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:50:02.624055    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39884c94_9c25_4e4f_b2ff_f4378fb16a7b.slice/crio-533c0bdaf5389ee52b44f730d3f8ad443d69a0bf6496dd2cb9150367bb027cfa WatchSource:0}: Error finding container 533c0bdaf5389ee52b44f730d3f8ad443d69a0bf6496dd2cb9150367bb027cfa: Status 404 returned error can't find the container with id 533c0bdaf5389ee52b44f730d3f8ad443d69a0bf6496dd2cb9150367bb027cfa
Oct 31 03:50:02.625158 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:02.625134    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7153-bf2e87"]
Oct 31 03:50:02.822494 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:02.822462    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb" event={"ID":"76378008-52e0-44da-9bf4-7c6745178755","Type":"ContainerStarted","Data":"96e50bd548219fc976f2ed3c77c618c43f17a9701865cb558e52bc23a4ddd067"}
Oct 31 03:50:02.822792 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:02.822520    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb" event={"ID":"76378008-52e0-44da-9bf4-7c6745178755","Type":"ContainerStarted","Data":"b8d44e98fbaeb72fb89bdbf8e48cd68377d7288ecb9a5e41b2f799b6febb3a21"}
Oct 31 03:50:02.822792 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:02.822571    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb"
Oct 31 03:50:02.824206 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:02.824180    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7153-bf2e87" event={"ID":"39884c94-9c25-4e4f-b2ff-f4378fb16a7b","Type":"ContainerStarted","Data":"2dd0640c94529b5c4c8ad4753d2497c0a3fadc891cb8a0f047dfc1f81f6ff496"}
Oct 31 03:50:02.824322 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:02.824220    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7153-bf2e87" event={"ID":"39884c94-9c25-4e4f-b2ff-f4378fb16a7b","Type":"ContainerStarted","Data":"533c0bdaf5389ee52b44f730d3f8ad443d69a0bf6496dd2cb9150367bb027cfa"}
Oct 31 03:50:02.825977 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:02.825950    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx" event={"ID":"f334da75-10b1-4320-9d31-4b8d428040d6","Type":"ContainerStarted","Data":"893b776234ab7089837dcfe4e1fd130afee3bc55f6776dc9a02e7d422bd9fb19"}
Oct 31 03:50:02.826064 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:02.826000    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx" event={"ID":"f334da75-10b1-4320-9d31-4b8d428040d6","Type":"ContainerStarted","Data":"4ed6d2c566b23a6e33cdd39f89d0c4f62a8a745bce4e4eb81e95e35c20cda8dd"}
Oct 31 03:50:02.826064 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:02.826032    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx"
Oct 31 03:50:02.939767 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:02.939702    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb" podStartSLOduration=1.939682371 podStartE2EDuration="1.939682371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:02.938159943 +0000 UTC m=+1828.449142320" watchObservedRunningTime="2026-10-31 03:50:02.939682371 +0000 UTC m=+1828.450664749"
Oct 31 03:50:03.059526 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:03.059474    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7153-bf2e87" podStartSLOduration=2.059457071 podStartE2EDuration="2.059457071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:03.051731551 +0000 UTC m=+1828.562713928" watchObservedRunningTime="2026-10-31 03:50:03.059457071 +0000 UTC m=+1828.570439445"
Oct 31 03:50:03.174164 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:03.174062    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx" podStartSLOduration=2.174043963 podStartE2EDuration="2.174043963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:03.170363476 +0000 UTC m=+1828.681345854" watchObservedRunningTime="2026-10-31 03:50:03.174043963 +0000 UTC m=+1828.685026340"
Oct 31 03:50:04.547993 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.547956    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7153-bf2e87"]
Oct 31 03:50:04.553777 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.553745    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7153-bf2e87"]
Oct 31 03:50:04.715017 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.714979    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7153-e1fcb6"]
Oct 31 03:50:04.716415 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.716384    4447 cpu_manager.go:414] "containerMap: removing container" podUID="39884c94-9c25-4e4f-b2ff-f4378fb16a7b" containerName="test-curl-7153-bf2e87"
Oct 31 03:50:04.716415 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.716415    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39884c94-9c25-4e4f-b2ff-f4378fb16a7b" containerName="test-curl-7153-bf2e87"
Oct 31 03:50:04.716614 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.716596    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="39884c94-9c25-4e4f-b2ff-f4378fb16a7b" containerName="test-curl-7153-bf2e87"
Oct 31 03:50:04.721411 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.721333    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7153-e1fcb6"
Oct 31 03:50:04.771642 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.771610    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7153-e1fcb6"]
Oct 31 03:50:04.817624 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.817599    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:50:04.821209 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.821186    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzf4g\" (UniqueName: \"kubernetes.io/projected/dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b-kube-api-access-gzf4g\") pod \"test-curl-7153-e1fcb6\" (UID: \"dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b\") " pod="opendatahub/test-curl-7153-e1fcb6"
Oct 31 03:50:04.845336 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.845307    4447 generic.go:356] "Generic (PLEG): container finished" podID="377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f" containerID="bb0be6b368ff75326103c6413b3f785eeafcc65f4730fe6e5793d46baa2961a2" exitCode=0
Oct 31 03:50:04.845471 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.845352    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"
Oct 31 03:50:04.845471 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.845396    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs" event={"ID":"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f","Type":"ContainerDied","Data":"bb0be6b368ff75326103c6413b3f785eeafcc65f4730fe6e5793d46baa2961a2"}
Oct 31 03:50:04.845471 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.845456    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs" event={"ID":"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f","Type":"ContainerDied","Data":"acb310576d31ca501f6c434b923cdfbbe2a870f6e3e8192f32d3e2904e3c5838"}
Oct 31 03:50:04.845593 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.845487    4447 scope.go:122] "RemoveContainer" containerID="bb0be6b368ff75326103c6413b3f785eeafcc65f4730fe6e5793d46baa2961a2"
Oct 31 03:50:04.845593 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.845517    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7153-bf2e87" podUID="39884c94-9c25-4e4f-b2ff-f4378fb16a7b" containerName="test-curl-7153-bf2e87" containerID="cri-o://2dd0640c94529b5c4c8ad4753d2497c0a3fadc891cb8a0f047dfc1f81f6ff496" gracePeriod=2
Oct 31 03:50:04.922455 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.922420    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-credential-socket\") pod \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") "
Oct 31 03:50:04.922638 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.922535    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-token\" (UniqueName: \"kubernetes.io/projected/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-token\") pod \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") "
Oct 31 03:50:04.922638 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.922574    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-data\") pod \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") "
Oct 31 03:50:04.922638 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.922615    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-kube-api-access-r4bzb\" (UniqueName: \"kubernetes.io/projected/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-kube-api-access-r4bzb\") pod \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") "
Oct 31 03:50:04.922638 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.922634    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-envoy\") pod \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") "
Oct 31 03:50:04.922823 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.922698    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istiod-ca-cert\") pod \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") "
Oct 31 03:50:04.922823 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.922698    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-credential-socket" pod "377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f" (UID: "377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:04.922823 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.922736    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-workload-certs\") pod \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") "
Oct 31 03:50:04.923000 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.922869    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-data" pod "377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f" (UID: "377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:04.923000 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.922932    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-podinfo\") pod \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") "
Oct 31 03:50:04.923000 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.922968    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-workload-socket\") pod \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\" (UID: \"377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f\") "
Oct 31 03:50:04.923163 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.923071    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-workload-certs" pod "377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f" (UID: "377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:04.923163 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.923147    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istiod-ca-cert" pod "377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f" (UID: "377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:50:04.923316 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.923294    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzf4g\" (UniqueName: \"kubernetes.io/projected/dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b-kube-api-access-gzf4g\") pod \"test-curl-7153-e1fcb6\" (UID: \"dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b\") " pod="opendatahub/test-curl-7153-e1fcb6"
Oct 31 03:50:04.923538 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.923467    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-workload-socket" pod "377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f" (UID: "377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:04.923847 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.923826    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:04.923847 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.923848    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:04.924045 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.923857    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:04.924045 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.923867    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:04.924045 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.923875    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:04.925177 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.925152    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-envoy" pod "377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f" (UID: "377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:04.925177 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.925170    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-kube-api-access-r4bzb" pod "377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f" (UID: "377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f"). InnerVolumeSpecName "kube-api-access-r4bzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:04.925498 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.925478    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-podinfo" pod "377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f" (UID: "377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:50:04.925553 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.925512    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-token" pod "377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f" (UID: "377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:04.949064 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.949043    4447 scope.go:122] "RemoveContainer" containerID="bb0be6b368ff75326103c6413b3f785eeafcc65f4730fe6e5793d46baa2961a2"
Oct 31 03:50:04.949459 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:50:04.949435    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb0be6b368ff75326103c6413b3f785eeafcc65f4730fe6e5793d46baa2961a2\": container with ID starting with bb0be6b368ff75326103c6413b3f785eeafcc65f4730fe6e5793d46baa2961a2 not found: ID does not exist" containerID="bb0be6b368ff75326103c6413b3f785eeafcc65f4730fe6e5793d46baa2961a2"
Oct 31 03:50:04.949523 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.949471    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0be6b368ff75326103c6413b3f785eeafcc65f4730fe6e5793d46baa2961a2"} err="failed to get container status \"bb0be6b368ff75326103c6413b3f785eeafcc65f4730fe6e5793d46baa2961a2\": rpc error: code = NotFound desc = could not find container \"bb0be6b368ff75326103c6413b3f785eeafcc65f4730fe6e5793d46baa2961a2\": container with ID starting with bb0be6b368ff75326103c6413b3f785eeafcc65f4730fe6e5793d46baa2961a2 not found: ID does not exist"
Oct 31 03:50:04.990954 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:04.989189    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzf4g\" (UniqueName: \"kubernetes.io/projected/dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b-kube-api-access-gzf4g\") pod \"test-curl-7153-e1fcb6\" (UID: \"dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b\") " pod="opendatahub/test-curl-7153-e1fcb6"
Oct 31 03:50:05.026043 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.025761    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.026043 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.025797    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4bzb\" (UniqueName: \"kubernetes.io/projected/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-kube-api-access-r4bzb\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.026043 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.025811    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.026043 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.025821    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.042876 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.042801    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7153-e1fcb6"
Oct 31 03:50:05.110990 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.109808    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:50:05.229101 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.229074    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-credential-socket\") pod \"29f09a6c-204d-410b-b765-65368fd42c9d\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") "
Oct 31 03:50:05.229213 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.229145    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29f09a6c-204d-410b-b765-65368fd42c9d-kube-api-access-6mphr\" (UniqueName: \"kubernetes.io/projected/29f09a6c-204d-410b-b765-65368fd42c9d-kube-api-access-6mphr\") pod \"29f09a6c-204d-410b-b765-65368fd42c9d\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") "
Oct 31 03:50:05.229301 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.229285    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29f09a6c-204d-410b-b765-65368fd42c9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29f09a6c-204d-410b-b765-65368fd42c9d-istiod-ca-cert\") pod \"29f09a6c-204d-410b-b765-65368fd42c9d\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") "
Oct 31 03:50:05.229344 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.229334    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-workload-socket\") pod \"29f09a6c-204d-410b-b765-65368fd42c9d\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") "
Oct 31 03:50:05.229483 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.229399    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-credential-socket" pod "29f09a6c-204d-410b-b765-65368fd42c9d" (UID: "29f09a6c-204d-410b-b765-65368fd42c9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:05.229483 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.229404    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29f09a6c-204d-410b-b765-65368fd42c9d-istio-token\" (UniqueName: \"kubernetes.io/projected/29f09a6c-204d-410b-b765-65368fd42c9d-istio-token\") pod \"29f09a6c-204d-410b-b765-65368fd42c9d\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") "
Oct 31 03:50:05.229625 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.229604    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-workload-certs\") pod \"29f09a6c-204d-410b-b765-65368fd42c9d\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") "
Oct 31 03:50:05.229687 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.229647    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29f09a6c-204d-410b-b765-65368fd42c9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29f09a6c-204d-410b-b765-65368fd42c9d-istio-podinfo\") pod \"29f09a6c-204d-410b-b765-65368fd42c9d\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") "
Oct 31 03:50:05.229746 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.229679    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-workload-socket" pod "29f09a6c-204d-410b-b765-65368fd42c9d" (UID: "29f09a6c-204d-410b-b765-65368fd42c9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:05.229746 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.229678    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29f09a6c-204d-410b-b765-65368fd42c9d-istiod-ca-cert" pod "29f09a6c-204d-410b-b765-65368fd42c9d" (UID: "29f09a6c-204d-410b-b765-65368fd42c9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:50:05.229870 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.229851    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-istio-envoy\") pod \"29f09a6c-204d-410b-b765-65368fd42c9d\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") "
Oct 31 03:50:05.229958 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.229941    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-istio-data\") pod \"29f09a6c-204d-410b-b765-65368fd42c9d\" (UID: \"29f09a6c-204d-410b-b765-65368fd42c9d\") "
Oct 31 03:50:05.230043 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.230021    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-workload-certs" pod "29f09a6c-204d-410b-b765-65368fd42c9d" (UID: "29f09a6c-204d-410b-b765-65368fd42c9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:05.230277 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.230260    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-istio-data" pod "29f09a6c-204d-410b-b765-65368fd42c9d" (UID: "29f09a6c-204d-410b-b765-65368fd42c9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:05.230985 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.230961    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.231099 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.230991    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.231099 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.231005    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29f09a6c-204d-410b-b765-65368fd42c9d-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.231099 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.231019    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.231099 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.231034    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.239517 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.239486    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29f09a6c-204d-410b-b765-65368fd42c9d-istio-token" pod "29f09a6c-204d-410b-b765-65368fd42c9d" (UID: "29f09a6c-204d-410b-b765-65368fd42c9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:05.250472 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.250320    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29f09a6c-204d-410b-b765-65368fd42c9d-kube-api-access-6mphr" pod "29f09a6c-204d-410b-b765-65368fd42c9d" (UID: "29f09a6c-204d-410b-b765-65368fd42c9d"). InnerVolumeSpecName "kube-api-access-6mphr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:05.250472 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.250322    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29f09a6c-204d-410b-b765-65368fd42c9d-istio-podinfo" pod "29f09a6c-204d-410b-b765-65368fd42c9d" (UID: "29f09a6c-204d-410b-b765-65368fd42c9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:50:05.250472 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.250380    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-istio-envoy" pod "29f09a6c-204d-410b-b765-65368fd42c9d" (UID: "29f09a6c-204d-410b-b765-65368fd42c9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:05.332736 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.332614    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29f09a6c-204d-410b-b765-65368fd42c9d-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.332736 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.332646    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29f09a6c-204d-410b-b765-65368fd42c9d-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.332736 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.332655    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mphr\" (UniqueName: \"kubernetes.io/projected/29f09a6c-204d-410b-b765-65368fd42c9d-kube-api-access-6mphr\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.332736 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.332665    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29f09a6c-204d-410b-b765-65368fd42c9d-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.368354 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.368284    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"]
Oct 31 03:50:05.390380 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.390332    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-hjccs"]
Oct 31 03:50:05.472179 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.472149    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7153-e1fcb6"]
Oct 31 03:50:05.473506 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:50:05.473477    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbd6e8a1_6f62_43ec_827e_d5bad9f6c29b.slice/crio-4d2b7ba9cfa7443281c98148a1f117a3830596b93aa8d0f35136f88d0e46deb9 WatchSource:0}: Error finding container 4d2b7ba9cfa7443281c98148a1f117a3830596b93aa8d0f35136f88d0e46deb9: Status 404 returned error can't find the container with id 4d2b7ba9cfa7443281c98148a1f117a3830596b93aa8d0f35136f88d0e46deb9
Oct 31 03:50:05.484766 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.484679    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:50:05.538139 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.536219    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-credential-socket\") pod \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") "
Oct 31 03:50:05.538139 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.536270    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-token\" (UniqueName: \"kubernetes.io/projected/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-token\") pod \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") "
Oct 31 03:50:05.538139 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.536455    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-workload-socket\") pod \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") "
Oct 31 03:50:05.538139 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.536489    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istiod-ca-cert\") pod \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") "
Oct 31 03:50:05.538139 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.536532    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-envoy\") pod \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") "
Oct 31 03:50:05.538139 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.536660    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-podinfo\") pod \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") "
Oct 31 03:50:05.538139 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.536696    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-kube-api-access-p6xc2\" (UniqueName: \"kubernetes.io/projected/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-kube-api-access-p6xc2\") pod \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") "
Oct 31 03:50:05.538139 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.536757    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-workload-certs\") pod \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") "
Oct 31 03:50:05.538139 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.536801    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-data\") pod \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\" (UID: \"d379c75c-04e8-4ab1-8b40-e8de8b0631ef\") "
Oct 31 03:50:05.539987 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.539640    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-data" pod "d379c75c-04e8-4ab1-8b40-e8de8b0631ef" (UID: "d379c75c-04e8-4ab1-8b40-e8de8b0631ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:05.540478 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.540150    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-credential-socket" pod "d379c75c-04e8-4ab1-8b40-e8de8b0631ef" (UID: "d379c75c-04e8-4ab1-8b40-e8de8b0631ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:05.542809 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.542779    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-token" pod "d379c75c-04e8-4ab1-8b40-e8de8b0631ef" (UID: "d379c75c-04e8-4ab1-8b40-e8de8b0631ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:05.546223 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.545943    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-podinfo" pod "d379c75c-04e8-4ab1-8b40-e8de8b0631ef" (UID: "d379c75c-04e8-4ab1-8b40-e8de8b0631ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:50:05.549930 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.547431    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-workload-certs" pod "d379c75c-04e8-4ab1-8b40-e8de8b0631ef" (UID: "d379c75c-04e8-4ab1-8b40-e8de8b0631ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:05.549930 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.547597    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-workload-socket" pod "d379c75c-04e8-4ab1-8b40-e8de8b0631ef" (UID: "d379c75c-04e8-4ab1-8b40-e8de8b0631ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:05.549930 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.547847    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istiod-ca-cert" pod "d379c75c-04e8-4ab1-8b40-e8de8b0631ef" (UID: "d379c75c-04e8-4ab1-8b40-e8de8b0631ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:50:05.554296 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.554269    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-envoy" pod "d379c75c-04e8-4ab1-8b40-e8de8b0631ef" (UID: "d379c75c-04e8-4ab1-8b40-e8de8b0631ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:05.563583 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.563555    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-kube-api-access-p6xc2" pod "d379c75c-04e8-4ab1-8b40-e8de8b0631ef" (UID: "d379c75c-04e8-4ab1-8b40-e8de8b0631ef"). InnerVolumeSpecName "kube-api-access-p6xc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:05.626147 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.626121    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:50:05.638755 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.638723    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.638755 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.638756    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6xc2\" (UniqueName: \"kubernetes.io/projected/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-kube-api-access-p6xc2\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.638998 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.638772    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.638998 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.638787    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.638998 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.638799    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.638998 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.638810    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.638998 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.638823    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.638998 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.638835    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.638998 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.638846    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d379c75c-04e8-4ab1-8b40-e8de8b0631ef-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.740604 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.740566    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-envoy\") pod \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") "
Oct 31 03:50:05.740757 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.740675    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-kube-api-access-5qqkr\" (UniqueName: \"kubernetes.io/projected/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-kube-api-access-5qqkr\") pod \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") "
Oct 31 03:50:05.740757 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.740725    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-workload-certs\") pod \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") "
Oct 31 03:50:05.740757 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.740752    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istiod-ca-cert\") pod \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") "
Oct 31 03:50:05.740928 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.740787    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-workload-socket\") pod \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") "
Oct 31 03:50:05.740928 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.740810    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-credential-socket\") pod \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") "
Oct 31 03:50:05.740928 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.740862    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-token\" (UniqueName: \"kubernetes.io/projected/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-token\") pod \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") "
Oct 31 03:50:05.741039 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.741020    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-data\") pod \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") "
Oct 31 03:50:05.741176 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.741159    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-podinfo\") pod \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\" (UID: \"24d2a84e-6aad-4669-a456-ebe20cf2f0e9\") "
Oct 31 03:50:05.741242 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.741191    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-workload-certs" pod "24d2a84e-6aad-4669-a456-ebe20cf2f0e9" (UID: "24d2a84e-6aad-4669-a456-ebe20cf2f0e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:05.741538 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.741513    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istiod-ca-cert" pod "24d2a84e-6aad-4669-a456-ebe20cf2f0e9" (UID: "24d2a84e-6aad-4669-a456-ebe20cf2f0e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:50:05.742057 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.742030    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.742169 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.742062    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.742169 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.741705    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-workload-socket" pod "24d2a84e-6aad-4669-a456-ebe20cf2f0e9" (UID: "24d2a84e-6aad-4669-a456-ebe20cf2f0e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:05.742385 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.742363    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-data" pod "24d2a84e-6aad-4669-a456-ebe20cf2f0e9" (UID: "24d2a84e-6aad-4669-a456-ebe20cf2f0e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:05.742526 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.742381    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-credential-socket" pod "24d2a84e-6aad-4669-a456-ebe20cf2f0e9" (UID: "24d2a84e-6aad-4669-a456-ebe20cf2f0e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:05.743791 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.743767    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-kube-api-access-5qqkr" pod "24d2a84e-6aad-4669-a456-ebe20cf2f0e9" (UID: "24d2a84e-6aad-4669-a456-ebe20cf2f0e9"). InnerVolumeSpecName "kube-api-access-5qqkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:05.744114 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.744090    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-envoy" pod "24d2a84e-6aad-4669-a456-ebe20cf2f0e9" (UID: "24d2a84e-6aad-4669-a456-ebe20cf2f0e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:05.744344 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.744318    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-token" pod "24d2a84e-6aad-4669-a456-ebe20cf2f0e9" (UID: "24d2a84e-6aad-4669-a456-ebe20cf2f0e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:05.744798 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.744777    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-podinfo" pod "24d2a84e-6aad-4669-a456-ebe20cf2f0e9" (UID: "24d2a84e-6aad-4669-a456-ebe20cf2f0e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:50:05.844356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.844316    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.844356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.844357    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.844559 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.844371    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.844559 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.844385    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qqkr\" (UniqueName: \"kubernetes.io/projected/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-kube-api-access-5qqkr\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.844559 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.844399    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.844559 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.844414    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.844559 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.844427    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24d2a84e-6aad-4669-a456-ebe20cf2f0e9-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:05.874175 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.873946    4447 generic.go:356] "Generic (PLEG): container finished" podID="d379c75c-04e8-4ab1-8b40-e8de8b0631ef" containerID="b698f8298b01106fb48ff2ff22b84a4a816f68fb1099fee487f1c1bb14730390" exitCode=0
Oct 31 03:50:05.874175 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.874024    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"
Oct 31 03:50:05.874175 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.874035    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz" event={"ID":"d379c75c-04e8-4ab1-8b40-e8de8b0631ef","Type":"ContainerDied","Data":"b698f8298b01106fb48ff2ff22b84a4a816f68fb1099fee487f1c1bb14730390"}
Oct 31 03:50:05.874175 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.874134    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz" event={"ID":"d379c75c-04e8-4ab1-8b40-e8de8b0631ef","Type":"ContainerDied","Data":"76c5dd4e58678ac2091d1848f72e290c9e74d3f165d91612bb51a2fea8207cb5"}
Oct 31 03:50:05.874175 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.874167    4447 scope.go:122] "RemoveContainer" containerID="b698f8298b01106fb48ff2ff22b84a4a816f68fb1099fee487f1c1bb14730390"
Oct 31 03:50:05.877276 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.877254    4447 generic.go:356] "Generic (PLEG): container finished" podID="f1e5b912-f46c-4ce3-ad7e-521e506e5e88" containerID="0d37ad73241bf95ccbc852fe988a0f6bc29dfd5e747bf6ac8faf70d99b6077d2" exitCode=0
Oct 31 03:50:05.877401 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.877382    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5" event={"ID":"f1e5b912-f46c-4ce3-ad7e-521e506e5e88","Type":"ContainerDied","Data":"0d37ad73241bf95ccbc852fe988a0f6bc29dfd5e747bf6ac8faf70d99b6077d2"}
Oct 31 03:50:05.880941 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.880845    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7153-e1fcb6" event={"ID":"dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b","Type":"ContainerStarted","Data":"db89c21408d02707f22dedbd4e7200057e6d11e59da82752dd045a751331d3f6"}
Oct 31 03:50:05.880941 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.880910    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7153-e1fcb6" event={"ID":"dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b","Type":"ContainerStarted","Data":"4d2b7ba9cfa7443281c98148a1f117a3830596b93aa8d0f35136f88d0e46deb9"}
Oct 31 03:50:05.883438 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.883409    4447 generic.go:356] "Generic (PLEG): container finished" podID="29f09a6c-204d-410b-b765-65368fd42c9d" containerID="126c377ab7e035e92301e97bf83d38983e2d93d1ace3e133925ce0a5852dbc44" exitCode=0
Oct 31 03:50:05.883544 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.883472    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"
Oct 31 03:50:05.883544 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.883509    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt" event={"ID":"29f09a6c-204d-410b-b765-65368fd42c9d","Type":"ContainerDied","Data":"126c377ab7e035e92301e97bf83d38983e2d93d1ace3e133925ce0a5852dbc44"}
Oct 31 03:50:05.885574 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.885512    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt" event={"ID":"29f09a6c-204d-410b-b765-65368fd42c9d","Type":"ContainerDied","Data":"7685258bf38b88958369462d25c788744d286cc03ebbaf8af40a278dd6c981e0"}
Oct 31 03:50:05.890975 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.886125    4447 generic.go:356] "Generic (PLEG): container finished" podID="24d2a84e-6aad-4669-a456-ebe20cf2f0e9" containerID="edc9f8388d56defcc6825e96a1fdd571b057cfbf9ebe3aecf80a184b2532f7ee" exitCode=0
Oct 31 03:50:05.890975 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.886188    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd" event={"ID":"24d2a84e-6aad-4669-a456-ebe20cf2f0e9","Type":"ContainerDied","Data":"edc9f8388d56defcc6825e96a1fdd571b057cfbf9ebe3aecf80a184b2532f7ee"}
Oct 31 03:50:05.890975 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.886226    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd" event={"ID":"24d2a84e-6aad-4669-a456-ebe20cf2f0e9","Type":"ContainerDied","Data":"255b21e0bf99a4ca1bd78b9f731f6b0170b0d5e8ed1bc05b5f537f9aae0608ef"}
Oct 31 03:50:05.890975 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.886314    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"
Oct 31 03:50:05.944747 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.944721    4447 scope.go:122] "RemoveContainer" containerID="b698f8298b01106fb48ff2ff22b84a4a816f68fb1099fee487f1c1bb14730390"
Oct 31 03:50:05.945236 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:50:05.945213    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b698f8298b01106fb48ff2ff22b84a4a816f68fb1099fee487f1c1bb14730390\": container with ID starting with b698f8298b01106fb48ff2ff22b84a4a816f68fb1099fee487f1c1bb14730390 not found: ID does not exist" containerID="b698f8298b01106fb48ff2ff22b84a4a816f68fb1099fee487f1c1bb14730390"
Oct 31 03:50:05.945361 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.945251    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b698f8298b01106fb48ff2ff22b84a4a816f68fb1099fee487f1c1bb14730390"} err="failed to get container status \"b698f8298b01106fb48ff2ff22b84a4a816f68fb1099fee487f1c1bb14730390\": rpc error: code = NotFound desc = could not find container \"b698f8298b01106fb48ff2ff22b84a4a816f68fb1099fee487f1c1bb14730390\": container with ID starting with b698f8298b01106fb48ff2ff22b84a4a816f68fb1099fee487f1c1bb14730390 not found: ID does not exist"
Oct 31 03:50:05.945361 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.945278    4447 scope.go:122] "RemoveContainer" containerID="126c377ab7e035e92301e97bf83d38983e2d93d1ace3e133925ce0a5852dbc44"
Oct 31 03:50:05.954707 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.954683    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:50:05.957475 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.957374    4447 scope.go:122] "RemoveContainer" containerID="126c377ab7e035e92301e97bf83d38983e2d93d1ace3e133925ce0a5852dbc44"
Oct 31 03:50:05.957722 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:50:05.957676    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"126c377ab7e035e92301e97bf83d38983e2d93d1ace3e133925ce0a5852dbc44\": container with ID starting with 126c377ab7e035e92301e97bf83d38983e2d93d1ace3e133925ce0a5852dbc44 not found: ID does not exist" containerID="126c377ab7e035e92301e97bf83d38983e2d93d1ace3e133925ce0a5852dbc44"
Oct 31 03:50:05.957861 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.957712    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"126c377ab7e035e92301e97bf83d38983e2d93d1ace3e133925ce0a5852dbc44"} err="failed to get container status \"126c377ab7e035e92301e97bf83d38983e2d93d1ace3e133925ce0a5852dbc44\": rpc error: code = NotFound desc = could not find container \"126c377ab7e035e92301e97bf83d38983e2d93d1ace3e133925ce0a5852dbc44\": container with ID starting with 126c377ab7e035e92301e97bf83d38983e2d93d1ace3e133925ce0a5852dbc44 not found: ID does not exist"
Oct 31 03:50:05.957861 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.957737    4447 scope.go:122] "RemoveContainer" containerID="edc9f8388d56defcc6825e96a1fdd571b057cfbf9ebe3aecf80a184b2532f7ee"
Oct 31 03:50:05.975097 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.974808    4447 scope.go:122] "RemoveContainer" containerID="edc9f8388d56defcc6825e96a1fdd571b057cfbf9ebe3aecf80a184b2532f7ee"
Oct 31 03:50:05.975230 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:50:05.975203    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edc9f8388d56defcc6825e96a1fdd571b057cfbf9ebe3aecf80a184b2532f7ee\": container with ID starting with edc9f8388d56defcc6825e96a1fdd571b057cfbf9ebe3aecf80a184b2532f7ee not found: ID does not exist" containerID="edc9f8388d56defcc6825e96a1fdd571b057cfbf9ebe3aecf80a184b2532f7ee"
Oct 31 03:50:05.975324 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:05.975235    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc9f8388d56defcc6825e96a1fdd571b057cfbf9ebe3aecf80a184b2532f7ee"} err="failed to get container status \"edc9f8388d56defcc6825e96a1fdd571b057cfbf9ebe3aecf80a184b2532f7ee\": rpc error: code = NotFound desc = could not find container \"edc9f8388d56defcc6825e96a1fdd571b057cfbf9ebe3aecf80a184b2532f7ee\": container with ID starting with edc9f8388d56defcc6825e96a1fdd571b057cfbf9ebe3aecf80a184b2532f7ee not found: ID does not exist"
Oct 31 03:50:06.046630 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.046569    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"]
Oct 31 03:50:06.046993 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.046336    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-kube-api-access-rfwrr\" (UniqueName: \"kubernetes.io/projected/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-kube-api-access-rfwrr\") pod \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") "
Oct 31 03:50:06.047153 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.047097    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-credential-socket\") pod \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") "
Oct 31 03:50:06.047153 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.047152    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-envoy\") pod \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") "
Oct 31 03:50:06.047282 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.047245    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-data\") pod \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") "
Oct 31 03:50:06.047282 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.047275    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-workload-certs\") pod \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") "
Oct 31 03:50:06.047386 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.047308    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-podinfo\") pod \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") "
Oct 31 03:50:06.047386 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.047369    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istiod-ca-cert\") pod \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") "
Oct 31 03:50:06.047516 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.047500    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-token\" (UniqueName: \"kubernetes.io/projected/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-token\") pod \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") "
Oct 31 03:50:06.047618 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.047603    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-workload-socket\") pod \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\" (UID: \"f1e5b912-f46c-4ce3-ad7e-521e506e5e88\") "
Oct 31 03:50:06.050646 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.048699    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-workload-certs" pod "f1e5b912-f46c-4ce3-ad7e-521e506e5e88" (UID: "f1e5b912-f46c-4ce3-ad7e-521e506e5e88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:06.050646 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.048972    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istiod-ca-cert" pod "f1e5b912-f46c-4ce3-ad7e-521e506e5e88" (UID: "f1e5b912-f46c-4ce3-ad7e-521e506e5e88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:50:06.050646 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.049103    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-credential-socket" pod "f1e5b912-f46c-4ce3-ad7e-521e506e5e88" (UID: "f1e5b912-f46c-4ce3-ad7e-521e506e5e88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:06.050646 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.049393    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-data" pod "f1e5b912-f46c-4ce3-ad7e-521e506e5e88" (UID: "f1e5b912-f46c-4ce3-ad7e-521e506e5e88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:06.050646 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.049767    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-kube-api-access-rfwrr" pod "f1e5b912-f46c-4ce3-ad7e-521e506e5e88" (UID: "f1e5b912-f46c-4ce3-ad7e-521e506e5e88"). InnerVolumeSpecName "kube-api-access-rfwrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:06.050646 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.050620    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-workload-socket" pod "f1e5b912-f46c-4ce3-ad7e-521e506e5e88" (UID: "f1e5b912-f46c-4ce3-ad7e-521e506e5e88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:06.051686 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.051656    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-token" pod "f1e5b912-f46c-4ce3-ad7e-521e506e5e88" (UID: "f1e5b912-f46c-4ce3-ad7e-521e506e5e88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:06.054054 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.054025    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-envoy" pod "f1e5b912-f46c-4ce3-ad7e-521e506e5e88" (UID: "f1e5b912-f46c-4ce3-ad7e-521e506e5e88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:06.054631 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.054608    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-podinfo" pod "f1e5b912-f46c-4ce3-ad7e-521e506e5e88" (UID: "f1e5b912-f46c-4ce3-ad7e-521e506e5e88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:50:06.056366 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.056342    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-klntz"]
Oct 31 03:50:06.150935 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.150861    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.151185 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.151157    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.151302 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.151186    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfwrr\" (UniqueName: \"kubernetes.io/projected/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-kube-api-access-rfwrr\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.151302 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.151223    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.151302 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.151238    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.151302 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.151252    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.151302 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.151266    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.151302 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.151298    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.151608 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.151313    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1e5b912-f46c-4ce3-ad7e-521e506e5e88-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.159309 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.155696    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:50:06.163259 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.163208    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7153-e1fcb6" podStartSLOduration=2.163195265 podStartE2EDuration="2.163195265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:06.160677864 +0000 UTC m=+1831.671660242" watchObservedRunningTime="2026-10-31 03:50:06.163195265 +0000 UTC m=+1831.674177641"
Oct 31 03:50:06.172849 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.172826    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:50:06.261577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.257612    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-workload-socket\") pod \"8954a5a4-2ec5-4443-adef-155c6781cb89\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") "
Oct 31 03:50:06.261577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.257754    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-podinfo\") pod \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") "
Oct 31 03:50:06.261577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.257848    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-data\") pod \"8954a5a4-2ec5-4443-adef-155c6781cb89\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") "
Oct 31 03:50:06.261577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.257910    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-envoy\") pod \"8954a5a4-2ec5-4443-adef-155c6781cb89\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") "
Oct 31 03:50:06.261577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.257989    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-token\" (UniqueName: \"kubernetes.io/projected/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-token\") pod \"8954a5a4-2ec5-4443-adef-155c6781cb89\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") "
Oct 31 03:50:06.261577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.258024    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-workload-certs\") pod \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") "
Oct 31 03:50:06.261577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.258059    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istiod-ca-cert\") pod \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") "
Oct 31 03:50:06.261577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.258099    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-envoy\") pod \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") "
Oct 31 03:50:06.261577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.258158    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-data\") pod \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") "
Oct 31 03:50:06.261577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.258192    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-workload-certs\") pod \"8954a5a4-2ec5-4443-adef-155c6781cb89\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") "
Oct 31 03:50:06.261577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.258287    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-podinfo\") pod \"8954a5a4-2ec5-4443-adef-155c6781cb89\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") "
Oct 31 03:50:06.261577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.258326    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-kube-api-access-tm7db\" (UniqueName: \"kubernetes.io/projected/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-kube-api-access-tm7db\") pod \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") "
Oct 31 03:50:06.261577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.258353    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8954a5a4-2ec5-4443-adef-155c6781cb89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8954a5a4-2ec5-4443-adef-155c6781cb89-istiod-ca-cert\") pod \"8954a5a4-2ec5-4443-adef-155c6781cb89\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") "
Oct 31 03:50:06.261577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.258385    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-token\" (UniqueName: \"kubernetes.io/projected/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-token\") pod \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") "
Oct 31 03:50:06.261577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.258413    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8954a5a4-2ec5-4443-adef-155c6781cb89-kube-api-access-7mcsp\" (UniqueName: \"kubernetes.io/projected/8954a5a4-2ec5-4443-adef-155c6781cb89-kube-api-access-7mcsp\") pod \"8954a5a4-2ec5-4443-adef-155c6781cb89\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") "
Oct 31 03:50:06.261577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.258503    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-workload-socket\") pod \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") "
Oct 31 03:50:06.261577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.258541    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-credential-socket\") pod \"8954a5a4-2ec5-4443-adef-155c6781cb89\" (UID: \"8954a5a4-2ec5-4443-adef-155c6781cb89\") "
Oct 31 03:50:06.261577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.258591    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-credential-socket\") pod \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\" (UID: \"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f\") "
Oct 31 03:50:06.265119 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.264919    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-credential-socket" pod "0926be7e-f12d-474d-a3c8-1a3eb3fbc43f" (UID: "0926be7e-f12d-474d-a3c8-1a3eb3fbc43f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:06.265473 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.265294    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-workload-socket" pod "8954a5a4-2ec5-4443-adef-155c6781cb89" (UID: "8954a5a4-2ec5-4443-adef-155c6781cb89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:06.265904 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.265604    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-data" pod "8954a5a4-2ec5-4443-adef-155c6781cb89" (UID: "8954a5a4-2ec5-4443-adef-155c6781cb89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:06.266414 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.266392    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.267968 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.267946    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.268096 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.268082    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.268182 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.267641    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istiod-ca-cert" pod "0926be7e-f12d-474d-a3c8-1a3eb3fbc43f" (UID: "0926be7e-f12d-474d-a3c8-1a3eb3fbc43f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:50:06.268182 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.267968    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-workload-socket" pod "0926be7e-f12d-474d-a3c8-1a3eb3fbc43f" (UID: "0926be7e-f12d-474d-a3c8-1a3eb3fbc43f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:06.269402 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.269355    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-workload-certs" pod "0926be7e-f12d-474d-a3c8-1a3eb3fbc43f" (UID: "0926be7e-f12d-474d-a3c8-1a3eb3fbc43f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:06.270407 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.270382    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-data" pod "0926be7e-f12d-474d-a3c8-1a3eb3fbc43f" (UID: "0926be7e-f12d-474d-a3c8-1a3eb3fbc43f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:06.271121 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.271051    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8954a5a4-2ec5-4443-adef-155c6781cb89-istiod-ca-cert" pod "8954a5a4-2ec5-4443-adef-155c6781cb89" (UID: "8954a5a4-2ec5-4443-adef-155c6781cb89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:50:06.271855 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.271776    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-workload-certs" pod "8954a5a4-2ec5-4443-adef-155c6781cb89" (UID: "8954a5a4-2ec5-4443-adef-155c6781cb89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:06.272133 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.272103    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-credential-socket" pod "8954a5a4-2ec5-4443-adef-155c6781cb89" (UID: "8954a5a4-2ec5-4443-adef-155c6781cb89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:06.273034 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.273000    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-kube-api-access-tm7db" pod "0926be7e-f12d-474d-a3c8-1a3eb3fbc43f" (UID: "0926be7e-f12d-474d-a3c8-1a3eb3fbc43f"). InnerVolumeSpecName "kube-api-access-tm7db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:06.281378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.281293    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-envoy" pod "0926be7e-f12d-474d-a3c8-1a3eb3fbc43f" (UID: "0926be7e-f12d-474d-a3c8-1a3eb3fbc43f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:06.283758 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.283709    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-token" pod "8954a5a4-2ec5-4443-adef-155c6781cb89" (UID: "8954a5a4-2ec5-4443-adef-155c6781cb89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:06.284014 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.283948    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"]
Oct 31 03:50:06.286773 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.286731    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8954a5a4-2ec5-4443-adef-155c6781cb89-kube-api-access-7mcsp" pod "8954a5a4-2ec5-4443-adef-155c6781cb89" (UID: "8954a5a4-2ec5-4443-adef-155c6781cb89"). InnerVolumeSpecName "kube-api-access-7mcsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:06.290239 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.290215    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-dhrmt"]
Oct 31 03:50:06.294613 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.294513    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-envoy" pod "8954a5a4-2ec5-4443-adef-155c6781cb89" (UID: "8954a5a4-2ec5-4443-adef-155c6781cb89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:06.294613 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.294571    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-podinfo" pod "0926be7e-f12d-474d-a3c8-1a3eb3fbc43f" (UID: "0926be7e-f12d-474d-a3c8-1a3eb3fbc43f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:50:06.296988 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.296944    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-token" pod "0926be7e-f12d-474d-a3c8-1a3eb3fbc43f" (UID: "0926be7e-f12d-474d-a3c8-1a3eb3fbc43f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:06.297687 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.297653    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-podinfo" pod "8954a5a4-2ec5-4443-adef-155c6781cb89" (UID: "8954a5a4-2ec5-4443-adef-155c6781cb89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:50:06.369901 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.369848    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.369901 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.369898    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.370111 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.369913    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.370111 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.369926    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.370111 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.369939    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.370111 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.369951    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.370111 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.369958    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.370111 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.369968    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.370111 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.369976    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8954a5a4-2ec5-4443-adef-155c6781cb89-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.370111 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.369985    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm7db\" (UniqueName: \"kubernetes.io/projected/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-kube-api-access-tm7db\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.370111 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.369994    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8954a5a4-2ec5-4443-adef-155c6781cb89-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.370111 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.370002    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.370111 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.370010    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mcsp\" (UniqueName: \"kubernetes.io/projected/8954a5a4-2ec5-4443-adef-155c6781cb89-kube-api-access-7mcsp\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.370111 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.370019    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.370111 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.370029    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8954a5a4-2ec5-4443-adef-155c6781cb89-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:06.396512 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.396424    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"]
Oct 31 03:50:06.401845 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.401816    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-cm7wd"]
Oct 31 03:50:06.615541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.615501    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24d2a84e-6aad-4669-a456-ebe20cf2f0e9" path="/var/lib/kubelet/pods/24d2a84e-6aad-4669-a456-ebe20cf2f0e9/volumes"
Oct 31 03:50:06.616290 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.616268    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29f09a6c-204d-410b-b765-65368fd42c9d" path="/var/lib/kubelet/pods/29f09a6c-204d-410b-b765-65368fd42c9d/volumes"
Oct 31 03:50:06.616755 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.616739    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f" path="/var/lib/kubelet/pods/377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f/volumes"
Oct 31 03:50:06.617388 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.617365    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d379c75c-04e8-4ab1-8b40-e8de8b0631ef" path="/var/lib/kubelet/pods/d379c75c-04e8-4ab1-8b40-e8de8b0631ef/volumes"
Oct 31 03:50:06.946219 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.946122    4447 generic.go:356] "Generic (PLEG): container finished" podID="0926be7e-f12d-474d-a3c8-1a3eb3fbc43f" containerID="aa90022d05a77afcadb5f852f4aa8d24d57b78805cfeb9c03ac6cd7d3bfe2b50" exitCode=0
Oct 31 03:50:06.946484 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.946460    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q" event={"ID":"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f","Type":"ContainerDied","Data":"aa90022d05a77afcadb5f852f4aa8d24d57b78805cfeb9c03ac6cd7d3bfe2b50"}
Oct 31 03:50:06.947357 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.946684    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q" event={"ID":"0926be7e-f12d-474d-a3c8-1a3eb3fbc43f","Type":"ContainerDied","Data":"17bd92b40f86a927b26b4a8b29768cc4ea2d8700376f6cadca49a1eb7d232493"}
Oct 31 03:50:06.956907 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.947055    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"
Oct 31 03:50:06.956907 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.947796    4447 scope.go:122] "RemoveContainer" containerID="aa90022d05a77afcadb5f852f4aa8d24d57b78805cfeb9c03ac6cd7d3bfe2b50"
Oct 31 03:50:06.959843 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.957545    4447 generic.go:356] "Generic (PLEG): container finished" podID="8954a5a4-2ec5-4443-adef-155c6781cb89" containerID="4846319f4401a761790dee40aa327c47d9374d74ee3f16042483df9ef8fe3760" exitCode=0
Oct 31 03:50:06.959843 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.957658    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m" event={"ID":"8954a5a4-2ec5-4443-adef-155c6781cb89","Type":"ContainerDied","Data":"4846319f4401a761790dee40aa327c47d9374d74ee3f16042483df9ef8fe3760"}
Oct 31 03:50:06.959843 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.957697    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m" event={"ID":"8954a5a4-2ec5-4443-adef-155c6781cb89","Type":"ContainerDied","Data":"1479185e901bda3c3b6040fb9b2863cb7e1786b469e12f8f5e97b516d4ece922"}
Oct 31 03:50:06.959843 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.957707    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"
Oct 31 03:50:06.960121 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.959877    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"
Oct 31 03:50:06.960121 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.959951    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5" event={"ID":"f1e5b912-f46c-4ce3-ad7e-521e506e5e88","Type":"ContainerDied","Data":"e1548d9a584aa6decb193adc88afc3a2002b7f2f057dbfb9b46dd111667ab8ef"}
Oct 31 03:50:06.981626 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.981604    4447 scope.go:122] "RemoveContainer" containerID="aa90022d05a77afcadb5f852f4aa8d24d57b78805cfeb9c03ac6cd7d3bfe2b50"
Oct 31 03:50:06.981975 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:50:06.981933    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa90022d05a77afcadb5f852f4aa8d24d57b78805cfeb9c03ac6cd7d3bfe2b50\": container with ID starting with aa90022d05a77afcadb5f852f4aa8d24d57b78805cfeb9c03ac6cd7d3bfe2b50 not found: ID does not exist" containerID="aa90022d05a77afcadb5f852f4aa8d24d57b78805cfeb9c03ac6cd7d3bfe2b50"
Oct 31 03:50:06.982027 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.981991    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa90022d05a77afcadb5f852f4aa8d24d57b78805cfeb9c03ac6cd7d3bfe2b50"} err="failed to get container status \"aa90022d05a77afcadb5f852f4aa8d24d57b78805cfeb9c03ac6cd7d3bfe2b50\": rpc error: code = NotFound desc = could not find container \"aa90022d05a77afcadb5f852f4aa8d24d57b78805cfeb9c03ac6cd7d3bfe2b50\": container with ID starting with aa90022d05a77afcadb5f852f4aa8d24d57b78805cfeb9c03ac6cd7d3bfe2b50 not found: ID does not exist"
Oct 31 03:50:06.982027 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.982015    4447 scope.go:122] "RemoveContainer" containerID="4846319f4401a761790dee40aa327c47d9374d74ee3f16042483df9ef8fe3760"
Oct 31 03:50:06.998378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.998353    4447 scope.go:122] "RemoveContainer" containerID="4846319f4401a761790dee40aa327c47d9374d74ee3f16042483df9ef8fe3760"
Oct 31 03:50:06.998716 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:50:06.998692    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4846319f4401a761790dee40aa327c47d9374d74ee3f16042483df9ef8fe3760\": container with ID starting with 4846319f4401a761790dee40aa327c47d9374d74ee3f16042483df9ef8fe3760 not found: ID does not exist" containerID="4846319f4401a761790dee40aa327c47d9374d74ee3f16042483df9ef8fe3760"
Oct 31 03:50:06.998761 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.998727    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4846319f4401a761790dee40aa327c47d9374d74ee3f16042483df9ef8fe3760"} err="failed to get container status \"4846319f4401a761790dee40aa327c47d9374d74ee3f16042483df9ef8fe3760\": rpc error: code = NotFound desc = could not find container \"4846319f4401a761790dee40aa327c47d9374d74ee3f16042483df9ef8fe3760\": container with ID starting with 4846319f4401a761790dee40aa327c47d9374d74ee3f16042483df9ef8fe3760 not found: ID does not exist"
Oct 31 03:50:06.998761 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:06.998746    4447 scope.go:122] "RemoveContainer" containerID="0d37ad73241bf95ccbc852fe988a0f6bc29dfd5e747bf6ac8faf70d99b6077d2"
Oct 31 03:50:07.121715 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:07.121251    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"]
Oct 31 03:50:07.129294 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:07.129253    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b98893-openshift-default-577d7f977c-h2xp5"]
Oct 31 03:50:07.219736 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:07.219711    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7153-bf2e87"
Oct 31 03:50:07.243221 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:07.243187    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"]
Oct 31 03:50:07.250486 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:07.250459    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-mwv5q"]
Oct 31 03:50:07.305668 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:07.305634    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39884c94-9c25-4e4f-b2ff-f4378fb16a7b-kube-api-access-l5j5v\" (UniqueName: \"kubernetes.io/projected/39884c94-9c25-4e4f-b2ff-f4378fb16a7b-kube-api-access-l5j5v\") pod \"39884c94-9c25-4e4f-b2ff-f4378fb16a7b\" (UID: \"39884c94-9c25-4e4f-b2ff-f4378fb16a7b\") "
Oct 31 03:50:07.307931 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:07.307870    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39884c94-9c25-4e4f-b2ff-f4378fb16a7b-kube-api-access-l5j5v" pod "39884c94-9c25-4e4f-b2ff-f4378fb16a7b" (UID: "39884c94-9c25-4e4f-b2ff-f4378fb16a7b"). InnerVolumeSpecName "kube-api-access-l5j5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:07.358616 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:07.358582    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"]
Oct 31 03:50:07.365767 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:07.365736    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-ff84m"]
Oct 31 03:50:07.408122 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:07.408043    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5j5v\" (UniqueName: \"kubernetes.io/projected/39884c94-9c25-4e4f-b2ff-f4378fb16a7b-kube-api-access-l5j5v\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:07.866579 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:07.866493    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7153-e1fcb6"]
Oct 31 03:50:07.870577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:07.868831    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7153-e1fcb6"]
Oct 31 03:50:07.971233 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:07.971195    4447 generic.go:356] "Generic (PLEG): container finished" podID="39884c94-9c25-4e4f-b2ff-f4378fb16a7b" containerID="2dd0640c94529b5c4c8ad4753d2497c0a3fadc891cb8a0f047dfc1f81f6ff496" exitCode=137
Oct 31 03:50:07.971440 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:07.971252    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7153-bf2e87"
Oct 31 03:50:07.971440 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:07.971280    4447 scope.go:122] "RemoveContainer" containerID="2dd0640c94529b5c4c8ad4753d2497c0a3fadc891cb8a0f047dfc1f81f6ff496"
Oct 31 03:50:07.972661 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:07.972479    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7153-e1fcb6" podUID="dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b" containerName="test-curl-7153-e1fcb6" containerID="cri-o://db89c21408d02707f22dedbd4e7200057e6d11e59da82752dd045a751331d3f6" gracePeriod=2
Oct 31 03:50:08.037647 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:50:08.037599    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7153-e1fcb6\" is forbidden: User \"system:node:ip-10-0-134-255.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-255.us-west-1.compute.internal' and this object" podUID="dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b" pod="opendatahub/test-curl-7153-e1fcb6"
Oct 31 03:50:08.049913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:08.048820    4447 scope.go:122] "RemoveContainer" containerID="2dd0640c94529b5c4c8ad4753d2497c0a3fadc891cb8a0f047dfc1f81f6ff496"
Oct 31 03:50:08.049913 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:50:08.049256    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd0640c94529b5c4c8ad4753d2497c0a3fadc891cb8a0f047dfc1f81f6ff496\": container with ID starting with 2dd0640c94529b5c4c8ad4753d2497c0a3fadc891cb8a0f047dfc1f81f6ff496 not found: ID does not exist" containerID="2dd0640c94529b5c4c8ad4753d2497c0a3fadc891cb8a0f047dfc1f81f6ff496"
Oct 31 03:50:08.049913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:08.049292    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd0640c94529b5c4c8ad4753d2497c0a3fadc891cb8a0f047dfc1f81f6ff496"} err="failed to get container status \"2dd0640c94529b5c4c8ad4753d2497c0a3fadc891cb8a0f047dfc1f81f6ff496\": rpc error: code = NotFound desc = could not find container \"2dd0640c94529b5c4c8ad4753d2497c0a3fadc891cb8a0f047dfc1f81f6ff496\": container with ID starting with 2dd0640c94529b5c4c8ad4753d2497c0a3fadc891cb8a0f047dfc1f81f6ff496 not found: ID does not exist"
Oct 31 03:50:08.089407 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:50:08.089349    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7153-bf2e87\" is forbidden: User \"system:node:ip-10-0-134-255.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-255.us-west-1.compute.internal' and this object" podUID="39884c94-9c25-4e4f-b2ff-f4378fb16a7b" pod="opendatahub/test-curl-7153-bf2e87"
Oct 31 03:50:08.140957 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:50:08.140825    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7153-e1fcb6\" is forbidden: User \"system:node:ip-10-0-134-255.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-255.us-west-1.compute.internal' and this object" podUID="dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b" pod="opendatahub/test-curl-7153-e1fcb6"
Oct 31 03:50:08.618652 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:08.618620    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0926be7e-f12d-474d-a3c8-1a3eb3fbc43f" path="/var/lib/kubelet/pods/0926be7e-f12d-474d-a3c8-1a3eb3fbc43f/volumes"
Oct 31 03:50:08.619219 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:08.619199    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39884c94-9c25-4e4f-b2ff-f4378fb16a7b" path="/var/lib/kubelet/pods/39884c94-9c25-4e4f-b2ff-f4378fb16a7b/volumes"
Oct 31 03:50:08.619702 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:08.619677    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8954a5a4-2ec5-4443-adef-155c6781cb89" path="/var/lib/kubelet/pods/8954a5a4-2ec5-4443-adef-155c6781cb89/volumes"
Oct 31 03:50:08.620583 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:08.620562    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1e5b912-f46c-4ce3-ad7e-521e506e5e88" path="/var/lib/kubelet/pods/f1e5b912-f46c-4ce3-ad7e-521e506e5e88/volumes"
Oct 31 03:50:10.263122 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:10.263057    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7153-e1fcb6"
Oct 31 03:50:10.378261 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:10.378232    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b-kube-api-access-gzf4g\" (UniqueName: \"kubernetes.io/projected/dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b-kube-api-access-gzf4g\") pod \"dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b\" (UID: \"dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b\") "
Oct 31 03:50:10.380818 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:10.380783    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b-kube-api-access-gzf4g" pod "dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b" (UID: "dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b"). InnerVolumeSpecName "kube-api-access-gzf4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:10.480409 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:10.480354    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzf4g\" (UniqueName: \"kubernetes.io/projected/dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b-kube-api-access-gzf4g\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:10.610526 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:10.610439    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b" path="/var/lib/kubelet/pods/dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b/volumes"
Oct 31 03:50:11.051668 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:11.051635    4447 generic.go:356] "Generic (PLEG): container finished" podID="dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b" containerID="db89c21408d02707f22dedbd4e7200057e6d11e59da82752dd045a751331d3f6" exitCode=137
Oct 31 03:50:11.051856 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:11.051673    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7153-e1fcb6"
Oct 31 03:50:11.051856 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:11.051745    4447 scope.go:122] "RemoveContainer" containerID="db89c21408d02707f22dedbd4e7200057e6d11e59da82752dd045a751331d3f6"
Oct 31 03:50:11.062031 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:11.062016    4447 scope.go:122] "RemoveContainer" containerID="db89c21408d02707f22dedbd4e7200057e6d11e59da82752dd045a751331d3f6"
Oct 31 03:50:11.062331 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:50:11.062310    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db89c21408d02707f22dedbd4e7200057e6d11e59da82752dd045a751331d3f6\": container with ID starting with db89c21408d02707f22dedbd4e7200057e6d11e59da82752dd045a751331d3f6 not found: ID does not exist" containerID="db89c21408d02707f22dedbd4e7200057e6d11e59da82752dd045a751331d3f6"
Oct 31 03:50:11.062414 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:11.062341    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db89c21408d02707f22dedbd4e7200057e6d11e59da82752dd045a751331d3f6"} err="failed to get container status \"db89c21408d02707f22dedbd4e7200057e6d11e59da82752dd045a751331d3f6\": rpc error: code = NotFound desc = could not find container \"db89c21408d02707f22dedbd4e7200057e6d11e59da82752dd045a751331d3f6\": container with ID starting with db89c21408d02707f22dedbd4e7200057e6d11e59da82752dd045a751331d3f6 not found: ID does not exist"
Oct 31 03:50:13.838195 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:13.838166    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb"
Oct 31 03:50:13.838601 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:13.838232    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx"
Oct 31 03:50:14.162713 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.162629    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7153-7aa397"]
Oct 31 03:50:14.163488 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.163471    4447 cpu_manager.go:414] "containerMap: removing container" podUID="24d2a84e-6aad-4669-a456-ebe20cf2f0e9" containerName="istio-proxy"
Oct 31 03:50:14.163548 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.163490    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24d2a84e-6aad-4669-a456-ebe20cf2f0e9" containerName="istio-proxy"
Oct 31 03:50:14.163548 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.163521    4447 cpu_manager.go:414] "containerMap: removing container" podUID="0926be7e-f12d-474d-a3c8-1a3eb3fbc43f" containerName="istio-proxy"
Oct 31 03:50:14.163548 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.163526    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0926be7e-f12d-474d-a3c8-1a3eb3fbc43f" containerName="istio-proxy"
Oct 31 03:50:14.163548 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.163535    4447 cpu_manager.go:414] "containerMap: removing container" podUID="f1e5b912-f46c-4ce3-ad7e-521e506e5e88" containerName="istio-proxy"
Oct 31 03:50:14.163548 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.163540    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1e5b912-f46c-4ce3-ad7e-521e506e5e88" containerName="istio-proxy"
Oct 31 03:50:14.163690 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.163556    4447 cpu_manager.go:414] "containerMap: removing container" podUID="377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f" containerName="istio-proxy"
Oct 31 03:50:14.163690 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.163561    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f" containerName="istio-proxy"
Oct 31 03:50:14.163690 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.163571    4447 cpu_manager.go:414] "containerMap: removing container" podUID="29f09a6c-204d-410b-b765-65368fd42c9d" containerName="istio-proxy"
Oct 31 03:50:14.163690 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.163575    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29f09a6c-204d-410b-b765-65368fd42c9d" containerName="istio-proxy"
Oct 31 03:50:14.163690 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.163585    4447 cpu_manager.go:414] "containerMap: removing container" podUID="8954a5a4-2ec5-4443-adef-155c6781cb89" containerName="istio-proxy"
Oct 31 03:50:14.163690 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.163590    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8954a5a4-2ec5-4443-adef-155c6781cb89" containerName="istio-proxy"
Oct 31 03:50:14.163690 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.163605    4447 cpu_manager.go:414] "containerMap: removing container" podUID="dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b" containerName="test-curl-7153-e1fcb6"
Oct 31 03:50:14.163690 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.163609    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b" containerName="test-curl-7153-e1fcb6"
Oct 31 03:50:14.163690 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.163620    4447 cpu_manager.go:414] "containerMap: removing container" podUID="d379c75c-04e8-4ab1-8b40-e8de8b0631ef" containerName="istio-proxy"
Oct 31 03:50:14.163690 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.163625    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d379c75c-04e8-4ab1-8b40-e8de8b0631ef" containerName="istio-proxy"
Oct 31 03:50:14.164000 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.163720    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="29f09a6c-204d-410b-b765-65368fd42c9d" containerName="istio-proxy"
Oct 31 03:50:14.164000 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.163727    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="8954a5a4-2ec5-4443-adef-155c6781cb89" containerName="istio-proxy"
Oct 31 03:50:14.164000 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.163735    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd6e8a1-6f62-43ec-827e-d5bad9f6c29b" containerName="test-curl-7153-e1fcb6"
Oct 31 03:50:14.164000 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.163742    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="d379c75c-04e8-4ab1-8b40-e8de8b0631ef" containerName="istio-proxy"
Oct 31 03:50:14.164000 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.163749    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="24d2a84e-6aad-4669-a456-ebe20cf2f0e9" containerName="istio-proxy"
Oct 31 03:50:14.164000 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.163760    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="0926be7e-f12d-474d-a3c8-1a3eb3fbc43f" containerName="istio-proxy"
Oct 31 03:50:14.164000 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.163765    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1e5b912-f46c-4ce3-ad7e-521e506e5e88" containerName="istio-proxy"
Oct 31 03:50:14.164000 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.163772    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="377bb5ad-a2ce-4d5a-a93d-bad2fc2dce3f" containerName="istio-proxy"
Oct 31 03:50:14.168775 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.168758    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7153-7aa397"
Oct 31 03:50:14.221399 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.221366    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7153-7aa397"]
Oct 31 03:50:14.246763 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.246724    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9p4v\" (UniqueName: \"kubernetes.io/projected/5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31-kube-api-access-d9p4v\") pod \"test-curl-7153-7aa397\" (UID: \"5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31\") " pod="opendatahub/test-curl-7153-7aa397"
Oct 31 03:50:14.257109 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.257080    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hppm7\""
Oct 31 03:50:14.348726 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.348695    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9p4v\" (UniqueName: \"kubernetes.io/projected/5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31-kube-api-access-d9p4v\") pod \"test-curl-7153-7aa397\" (UID: \"5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31\") " pod="opendatahub/test-curl-7153-7aa397"
Oct 31 03:50:14.410248 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.410212    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9p4v\" (UniqueName: \"kubernetes.io/projected/5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31-kube-api-access-d9p4v\") pod \"test-curl-7153-7aa397\" (UID: \"5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31\") " pod="opendatahub/test-curl-7153-7aa397"
Oct 31 03:50:14.481819 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.481784    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7153-7aa397"
Oct 31 03:50:14.865806 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:14.865775    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7153-7aa397"]
Oct 31 03:50:14.866203 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:50:14.866136    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a8f62ed_42ca_474d_9b7d_7c5ae9a7aa31.slice/crio-6d547dda1e6ce2aaf7d19a34b1a9dcf0742dba3b9a2f44b7c12726c094bf532a WatchSource:0}: Error finding container 6d547dda1e6ce2aaf7d19a34b1a9dcf0742dba3b9a2f44b7c12726c094bf532a: Status 404 returned error can't find the container with id 6d547dda1e6ce2aaf7d19a34b1a9dcf0742dba3b9a2f44b7c12726c094bf532a
Oct 31 03:50:15.094446 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:15.094407    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7153-7aa397" event={"ID":"5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31","Type":"ContainerStarted","Data":"2bc23390a3441c67f69b17cfd5971efdf0f1a4824dfb2fe44b1d41206cc8fd57"}
Oct 31 03:50:15.094686 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:15.094461    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7153-7aa397" event={"ID":"5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31","Type":"ContainerStarted","Data":"6d547dda1e6ce2aaf7d19a34b1a9dcf0742dba3b9a2f44b7c12726c094bf532a"}
Oct 31 03:50:15.215168 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:15.215111    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7153-7aa397" podStartSLOduration=1.215093683 podStartE2EDuration="1.215093683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:15.213490016 +0000 UTC m=+1840.724472395" watchObservedRunningTime="2026-10-31 03:50:15.215093683 +0000 UTC m=+1840.726076059"
Oct 31 03:50:16.909951 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:16.891870    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7153-7aa397"]
Oct 31 03:50:16.909951 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:16.906010    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7153-7aa397"]
Oct 31 03:50:16.958910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:16.947251    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599"]
Oct 31 03:50:16.958910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:16.948676    4447 cpu_manager.go:414] "containerMap: removing container" podUID="5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31" containerName="test-curl-7153-7aa397"
Oct 31 03:50:16.958910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:16.948714    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31" containerName="test-curl-7153-7aa397"
Oct 31 03:50:16.958910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:16.949078    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31" containerName="test-curl-7153-7aa397"
Oct 31 03:50:16.958910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:16.954512    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599"
Oct 31 03:50:17.010420 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:17.010390    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599"]
Oct 31 03:50:17.022072 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:17.022041    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-84e130\""
Oct 31 03:50:17.103653 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:17.103587    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs"]
Oct 31 03:50:17.122939 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:17.121160    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4q99\" (UniqueName: \"kubernetes.io/projected/cd0b36fc-225c-4452-8d77-0b28c1679c7a-kube-api-access-z4q99\") pod \"payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599\" (UID: \"cd0b36fc-225c-4452-8d77-0b28c1679c7a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599"
Oct 31 03:50:17.122939 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:17.121224    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0b36fc-225c-4452-8d77-0b28c1679c7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599\" (UID: \"cd0b36fc-225c-4452-8d77-0b28c1679c7a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599"
Oct 31 03:50:17.127987 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:17.127947    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs"
Oct 31 03:50:17.182008 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:17.180219    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7153-7aa397" podUID="5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31" containerName="test-curl-7153-7aa397" containerID="cri-o://2bc23390a3441c67f69b17cfd5971efdf0f1a4824dfb2fe44b1d41206cc8fd57" gracePeriod=2
Oct 31 03:50:17.210994 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:17.209367    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-84e130-dockercfg-fvrlr\""
Oct 31 03:50:17.223356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:17.223315    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqpnl\" (UniqueName: \"kubernetes.io/projected/c75320fc-6fe9-40e0-a67a-f70757ffd71a-kube-api-access-qqpnl\") pod \"payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs\" (UID: \"c75320fc-6fe9-40e0-a67a-f70757ffd71a\") " pod="openshift-ingress/payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs"
Oct 31 03:50:17.223679 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:17.223655    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75320fc-6fe9-40e0-a67a-f70757ffd71a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs\" (UID: \"c75320fc-6fe9-40e0-a67a-f70757ffd71a\") " pod="openshift-ingress/payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs"
Oct 31 03:50:17.223923 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:17.223871    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4q99\" (UniqueName: \"kubernetes.io/projected/cd0b36fc-225c-4452-8d77-0b28c1679c7a-kube-api-access-z4q99\") pod \"payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599\" (UID: \"cd0b36fc-225c-4452-8d77-0b28c1679c7a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599"
Oct 31 03:50:17.224075 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:17.224058    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0b36fc-225c-4452-8d77-0b28c1679c7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599\" (UID: \"cd0b36fc-225c-4452-8d77-0b28c1679c7a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599"
Oct 31 03:50:17.225175 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:17.225153    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0b36fc-225c-4452-8d77-0b28c1679c7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599\" (UID: \"cd0b36fc-225c-4452-8d77-0b28c1679c7a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599"
Oct 31 03:50:17.244637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:17.244569    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs"]
Oct 31 03:50:17.294116 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:17.294086    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4q99\" (UniqueName: \"kubernetes.io/projected/cd0b36fc-225c-4452-8d77-0b28c1679c7a-kube-api-access-z4q99\") pod \"payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599\" (UID: \"cd0b36fc-225c-4452-8d77-0b28c1679c7a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599"
Oct 31 03:50:17.309431 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:17.309401    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599"
Oct 31 03:50:17.325777 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:17.325740    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqpnl\" (UniqueName: \"kubernetes.io/projected/c75320fc-6fe9-40e0-a67a-f70757ffd71a-kube-api-access-qqpnl\") pod \"payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs\" (UID: \"c75320fc-6fe9-40e0-a67a-f70757ffd71a\") " pod="openshift-ingress/payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs"
Oct 31 03:50:17.325777 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:17.325783    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75320fc-6fe9-40e0-a67a-f70757ffd71a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs\" (UID: \"c75320fc-6fe9-40e0-a67a-f70757ffd71a\") " pod="openshift-ingress/payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs"
Oct 31 03:50:17.326704 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:17.326677    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75320fc-6fe9-40e0-a67a-f70757ffd71a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs\" (UID: \"c75320fc-6fe9-40e0-a67a-f70757ffd71a\") " pod="openshift-ingress/payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs"
Oct 31 03:50:17.342770 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:50:17.342718    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7153-7aa397\" is forbidden: User \"system:node:ip-10-0-134-255.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-255.us-west-1.compute.internal' and this object" podUID="5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31" pod="opendatahub/test-curl-7153-7aa397"
Oct 31 03:50:17.389728 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:17.389657    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqpnl\" (UniqueName: \"kubernetes.io/projected/c75320fc-6fe9-40e0-a67a-f70757ffd71a-kube-api-access-qqpnl\") pod \"payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs\" (UID: \"c75320fc-6fe9-40e0-a67a-f70757ffd71a\") " pod="openshift-ingress/payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs"
Oct 31 03:50:17.480035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:17.480001    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs"
Oct 31 03:50:17.696475 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:17.696434    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599"]
Oct 31 03:50:17.867473 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:50:17.867442    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75320fc_6fe9_40e0_a67a_f70757ffd71a.slice/crio-390562a946cec6c6f4544b52b43c147b906bf836e6942eb432383031db293d52 WatchSource:0}: Error finding container 390562a946cec6c6f4544b52b43c147b906bf836e6942eb432383031db293d52: Status 404 returned error can't find the container with id 390562a946cec6c6f4544b52b43c147b906bf836e6942eb432383031db293d52
Oct 31 03:50:17.868248 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:17.868225    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs"]
Oct 31 03:50:18.189943 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:18.189863    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs" event={"ID":"c75320fc-6fe9-40e0-a67a-f70757ffd71a","Type":"ContainerStarted","Data":"09c8f3468aef28571e01f18fd10acbbfc369bce1a6578ced807d56c321e18960"}
Oct 31 03:50:18.190454 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:18.190116    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs"
Oct 31 03:50:18.190454 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:18.190149    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs" event={"ID":"c75320fc-6fe9-40e0-a67a-f70757ffd71a","Type":"ContainerStarted","Data":"390562a946cec6c6f4544b52b43c147b906bf836e6942eb432383031db293d52"}
Oct 31 03:50:18.191878 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:18.191852    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599" event={"ID":"cd0b36fc-225c-4452-8d77-0b28c1679c7a","Type":"ContainerStarted","Data":"bf693d926f1d1cd6d2069cc0cfb09a5e184782f7ee0cae7f59475d9b5843df7e"}
Oct 31 03:50:18.192050 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:18.191953    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599" event={"ID":"cd0b36fc-225c-4452-8d77-0b28c1679c7a","Type":"ContainerStarted","Data":"435b3e1ea9246cc947d11de20b5e0a6da8b9301d6f5e7372c556bcc6392b55e6"}
Oct 31 03:50:18.192050 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:18.191997    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599"
Oct 31 03:50:18.309867 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:18.309765    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs" podStartSLOduration=1.309745992 podStartE2EDuration="1.309745992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:18.306345131 +0000 UTC m=+1843.817327508" watchObservedRunningTime="2026-10-31 03:50:18.309745992 +0000 UTC m=+1843.820728367"
Oct 31 03:50:18.357631 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:50:18.357589    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7153-7aa397\" is forbidden: User \"system:node:ip-10-0-134-255.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-255.us-west-1.compute.internal' and this object" podUID="5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31" pod="opendatahub/test-curl-7153-7aa397"
Oct 31 03:50:18.480318 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:18.480252    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599" podStartSLOduration=2.48023339 podStartE2EDuration="2.48023339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:18.472694241 +0000 UTC m=+1843.983676620" watchObservedRunningTime="2026-10-31 03:50:18.48023339 +0000 UTC m=+1843.991215835"
Oct 31 03:50:18.524045 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:50:18.524004    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7153-7aa397\" is forbidden: User \"system:node:ip-10-0-134-255.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-255.us-west-1.compute.internal' and this object" podUID="5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31" pod="opendatahub/test-curl-7153-7aa397"
Oct 31 03:50:19.004434 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:50:19.004399    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:50:19.456387 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:19.456354    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7153-7aa397"
Oct 31 03:50:19.560271 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:19.560180    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31-kube-api-access-d9p4v\" (UniqueName: \"kubernetes.io/projected/5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31-kube-api-access-d9p4v\") pod \"5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31\" (UID: \"5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31\") "
Oct 31 03:50:19.562554 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:19.562524    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31-kube-api-access-d9p4v" pod "5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31" (UID: "5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31"). InnerVolumeSpecName "kube-api-access-d9p4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:19.661808 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:19.661772    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9p4v\" (UniqueName: \"kubernetes.io/projected/5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31-kube-api-access-d9p4v\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:20.223139 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:20.223108    4447 generic.go:356] "Generic (PLEG): container finished" podID="5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31" containerID="2bc23390a3441c67f69b17cfd5971efdf0f1a4824dfb2fe44b1d41206cc8fd57" exitCode=137
Oct 31 03:50:20.223296 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:20.223152    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7153-7aa397"
Oct 31 03:50:20.223296 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:20.223210    4447 scope.go:122] "RemoveContainer" containerID="2bc23390a3441c67f69b17cfd5971efdf0f1a4824dfb2fe44b1d41206cc8fd57"
Oct 31 03:50:20.233931 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:20.233733    4447 scope.go:122] "RemoveContainer" containerID="2bc23390a3441c67f69b17cfd5971efdf0f1a4824dfb2fe44b1d41206cc8fd57"
Oct 31 03:50:20.234031 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:50:20.234015    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bc23390a3441c67f69b17cfd5971efdf0f1a4824dfb2fe44b1d41206cc8fd57\": container with ID starting with 2bc23390a3441c67f69b17cfd5971efdf0f1a4824dfb2fe44b1d41206cc8fd57 not found: ID does not exist" containerID="2bc23390a3441c67f69b17cfd5971efdf0f1a4824dfb2fe44b1d41206cc8fd57"
Oct 31 03:50:20.234071 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:20.234042    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bc23390a3441c67f69b17cfd5971efdf0f1a4824dfb2fe44b1d41206cc8fd57"} err="failed to get container status \"2bc23390a3441c67f69b17cfd5971efdf0f1a4824dfb2fe44b1d41206cc8fd57\": rpc error: code = NotFound desc = could not find container \"2bc23390a3441c67f69b17cfd5971efdf0f1a4824dfb2fe44b1d41206cc8fd57\": container with ID starting with 2bc23390a3441c67f69b17cfd5971efdf0f1a4824dfb2fe44b1d41206cc8fd57 not found: ID does not exist"
Oct 31 03:50:20.282503 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:50:20.282464    4447 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7153-7aa397\" is forbidden: User \"system:node:ip-10-0-134-255.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-255.us-west-1.compute.internal' and this object" podUID="5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31" pod="opendatahub/test-curl-7153-7aa397"
Oct 31 03:50:20.610981 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:20.610853    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31" path="/var/lib/kubelet/pods/5a8f62ed-42ca-474d-9b7d-7c5ae9a7aa31/volumes"
Oct 31 03:50:29.216363 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:29.216333    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs"
Oct 31 03:50:29.216773 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:29.216409    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599"
Oct 31 03:50:33.777979 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:33.777947    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw"]
Oct 31 03:50:33.890750 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:33.890689    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9"]
Oct 31 03:50:38.778834 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:38.778790    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw" podUID="2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc" containerName="payload-processing" containerID="cri-o://6c69ceb9a4af7433376316fdd083fcde1af88ced01f3e6aa818f03be075bbae7" gracePeriod=25
Oct 31 03:50:38.891527 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:38.891440    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9" podUID="8f6f0c1f-5401-4d66-abbf-1cc388649280" containerName="payload-pre-processing" containerID="cri-o://230c1e51fc57aa1fbfa09557eddebf3f4cf7e0e074a27cda0c962b69a2b6d9af" gracePeriod=25
Oct 31 03:50:39.081689 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.081660    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw"
Oct 31 03:50:39.161206 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.160749    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc-plugins-config-volume\") pod \"2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc\" (UID: \"2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc\") "
Oct 31 03:50:39.161206 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.161013    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc-kube-api-access-qcjtn\" (UniqueName: \"kubernetes.io/projected/2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc-kube-api-access-qcjtn\") pod \"2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc\" (UID: \"2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc\") "
Oct 31 03:50:39.161206 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.161156    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc-plugins-config-volume" pod "2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc" (UID: "2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:50:39.161648 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.161625    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:39.163709 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.163685    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc-kube-api-access-qcjtn" pod "2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc" (UID: "2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc"). InnerVolumeSpecName "kube-api-access-qcjtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:39.184926 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.184902    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9"
Oct 31 03:50:39.263000 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.262963    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f6f0c1f-5401-4d66-abbf-1cc388649280-kube-api-access-nk5kj\" (UniqueName: \"kubernetes.io/projected/8f6f0c1f-5401-4d66-abbf-1cc388649280-kube-api-access-nk5kj\") pod \"8f6f0c1f-5401-4d66-abbf-1cc388649280\" (UID: \"8f6f0c1f-5401-4d66-abbf-1cc388649280\") "
Oct 31 03:50:39.263206 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.263009    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f6f0c1f-5401-4d66-abbf-1cc388649280-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6f0c1f-5401-4d66-abbf-1cc388649280-plugins-config-volume\") pod \"8f6f0c1f-5401-4d66-abbf-1cc388649280\" (UID: \"8f6f0c1f-5401-4d66-abbf-1cc388649280\") "
Oct 31 03:50:39.263458 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.263429    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6f0c1f-5401-4d66-abbf-1cc388649280-plugins-config-volume" pod "8f6f0c1f-5401-4d66-abbf-1cc388649280" (UID: "8f6f0c1f-5401-4d66-abbf-1cc388649280"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:50:39.263570 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.263476    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcjtn\" (UniqueName: \"kubernetes.io/projected/2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc-kube-api-access-qcjtn\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:39.265224 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.265204    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f6f0c1f-5401-4d66-abbf-1cc388649280-kube-api-access-nk5kj" pod "8f6f0c1f-5401-4d66-abbf-1cc388649280" (UID: "8f6f0c1f-5401-4d66-abbf-1cc388649280"). InnerVolumeSpecName "kube-api-access-nk5kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:39.365009 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.364916    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6f0c1f-5401-4d66-abbf-1cc388649280-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:39.365009 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.364944    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk5kj\" (UniqueName: \"kubernetes.io/projected/8f6f0c1f-5401-4d66-abbf-1cc388649280-kube-api-access-nk5kj\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:39.383675 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.383645    4447 generic.go:356] "Generic (PLEG): container finished" podID="8f6f0c1f-5401-4d66-abbf-1cc388649280" containerID="230c1e51fc57aa1fbfa09557eddebf3f4cf7e0e074a27cda0c962b69a2b6d9af" exitCode=0
Oct 31 03:50:39.383857 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.383699    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9"
Oct 31 03:50:39.383857 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.383735    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9" event={"ID":"8f6f0c1f-5401-4d66-abbf-1cc388649280","Type":"ContainerDied","Data":"230c1e51fc57aa1fbfa09557eddebf3f4cf7e0e074a27cda0c962b69a2b6d9af"}
Oct 31 03:50:39.383857 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.383801    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9" event={"ID":"8f6f0c1f-5401-4d66-abbf-1cc388649280","Type":"ContainerDied","Data":"4563fff24432c7e5c2792c6f02b3839c5edd310458a4d4624d6367a6f158933a"}
Oct 31 03:50:39.383857 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.383833    4447 scope.go:122] "RemoveContainer" containerID="230c1e51fc57aa1fbfa09557eddebf3f4cf7e0e074a27cda0c962b69a2b6d9af"
Oct 31 03:50:39.385552 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.385532    4447 generic.go:356] "Generic (PLEG): container finished" podID="2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc" containerID="6c69ceb9a4af7433376316fdd083fcde1af88ced01f3e6aa818f03be075bbae7" exitCode=0
Oct 31 03:50:39.385668 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.385565    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw" event={"ID":"2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc","Type":"ContainerDied","Data":"6c69ceb9a4af7433376316fdd083fcde1af88ced01f3e6aa818f03be075bbae7"}
Oct 31 03:50:39.385668 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.385588    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw"
Oct 31 03:50:39.385668 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.385601    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw" event={"ID":"2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc","Type":"ContainerDied","Data":"f349149cd14a05ba35a26b574fe587fca6b435949e6b84b3ceb57487ebdc96b5"}
Oct 31 03:50:39.396859 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.396843    4447 scope.go:122] "RemoveContainer" containerID="230c1e51fc57aa1fbfa09557eddebf3f4cf7e0e074a27cda0c962b69a2b6d9af"
Oct 31 03:50:39.397183 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:50:39.397162    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"230c1e51fc57aa1fbfa09557eddebf3f4cf7e0e074a27cda0c962b69a2b6d9af\": container with ID starting with 230c1e51fc57aa1fbfa09557eddebf3f4cf7e0e074a27cda0c962b69a2b6d9af not found: ID does not exist" containerID="230c1e51fc57aa1fbfa09557eddebf3f4cf7e0e074a27cda0c962b69a2b6d9af"
Oct 31 03:50:39.397264 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.397194    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"230c1e51fc57aa1fbfa09557eddebf3f4cf7e0e074a27cda0c962b69a2b6d9af"} err="failed to get container status \"230c1e51fc57aa1fbfa09557eddebf3f4cf7e0e074a27cda0c962b69a2b6d9af\": rpc error: code = NotFound desc = could not find container \"230c1e51fc57aa1fbfa09557eddebf3f4cf7e0e074a27cda0c962b69a2b6d9af\": container with ID starting with 230c1e51fc57aa1fbfa09557eddebf3f4cf7e0e074a27cda0c962b69a2b6d9af not found: ID does not exist"
Oct 31 03:50:39.397264 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.397218    4447 scope.go:122] "RemoveContainer" containerID="6c69ceb9a4af7433376316fdd083fcde1af88ced01f3e6aa818f03be075bbae7"
Oct 31 03:50:39.412132 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.412110    4447 scope.go:122] "RemoveContainer" containerID="6c69ceb9a4af7433376316fdd083fcde1af88ced01f3e6aa818f03be075bbae7"
Oct 31 03:50:39.412441 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:50:39.412422    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c69ceb9a4af7433376316fdd083fcde1af88ced01f3e6aa818f03be075bbae7\": container with ID starting with 6c69ceb9a4af7433376316fdd083fcde1af88ced01f3e6aa818f03be075bbae7 not found: ID does not exist" containerID="6c69ceb9a4af7433376316fdd083fcde1af88ced01f3e6aa818f03be075bbae7"
Oct 31 03:50:39.412495 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.412451    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c69ceb9a4af7433376316fdd083fcde1af88ced01f3e6aa818f03be075bbae7"} err="failed to get container status \"6c69ceb9a4af7433376316fdd083fcde1af88ced01f3e6aa818f03be075bbae7\": rpc error: code = NotFound desc = could not find container \"6c69ceb9a4af7433376316fdd083fcde1af88ced01f3e6aa818f03be075bbae7\": container with ID starting with 6c69ceb9a4af7433376316fdd083fcde1af88ced01f3e6aa818f03be075bbae7 not found: ID does not exist"
Oct 31 03:50:39.554525 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.554493    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw"]
Oct 31 03:50:39.558819 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.558794    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e38e21-86967b667f-bxpsw"]
Oct 31 03:50:39.667605 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.667529    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9"]
Oct 31 03:50:39.674609 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:39.674581    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e38e21-786c77ff4c-cs5b9"]
Oct 31 03:50:40.612581 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:40.612543    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc" path="/var/lib/kubelet/pods/2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc/volumes"
Oct 31 03:50:40.613140 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:40.613123    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f6f0c1f-5401-4d66-abbf-1cc388649280" path="/var/lib/kubelet/pods/8f6f0c1f-5401-4d66-abbf-1cc388649280/volumes"
Oct 31 03:50:45.159320 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:45.159278    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"]
Oct 31 03:50:45.159798 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:45.159531    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8" podUID="57796b75-644c-40e6-9773-d143da5fb821" containerName="istio-proxy" containerID="cri-o://18ed63cae172bd167174279633990c21a6512ee34c0c714cd21da00ebdcb2e65" gracePeriod=30
Oct 31 03:50:49.004571 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:50:49.004533    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:50:50.482979 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.482593    4447 generic.go:356] "Generic (PLEG): container finished" podID="57796b75-644c-40e6-9773-d143da5fb821" containerID="18ed63cae172bd167174279633990c21a6512ee34c0c714cd21da00ebdcb2e65" exitCode=0
Oct 31 03:50:50.482979 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.482702    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8" event={"ID":"57796b75-644c-40e6-9773-d143da5fb821","Type":"ContainerDied","Data":"18ed63cae172bd167174279633990c21a6512ee34c0c714cd21da00ebdcb2e65"}
Oct 31 03:50:50.482979 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.482775    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8" event={"ID":"57796b75-644c-40e6-9773-d143da5fb821","Type":"ContainerDied","Data":"7a3d8d0eee7bb82c816aa55d2c0fa86c1db8d70d3c9d2dc3d0262daa39bb06be"}
Oct 31 03:50:50.482979 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.482792    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a3d8d0eee7bb82c816aa55d2c0fa86c1db8d70d3c9d2dc3d0262daa39bb06be"
Oct 31 03:50:50.486870 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.486846    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:50:50.593574 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.593541    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57796b75-644c-40e6-9773-d143da5fb821-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57796b75-644c-40e6-9773-d143da5fb821-istio-podinfo\") pod \"57796b75-644c-40e6-9773-d143da5fb821\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") "
Oct 31 03:50:50.593574 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.593579    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-workload-certs\") pod \"57796b75-644c-40e6-9773-d143da5fb821\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") "
Oct 31 03:50:50.593819 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.593691    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-credential-socket\") pod \"57796b75-644c-40e6-9773-d143da5fb821\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") "
Oct 31 03:50:50.593819 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.593757    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-istio-envoy\") pod \"57796b75-644c-40e6-9773-d143da5fb821\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") "
Oct 31 03:50:50.593819 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.593815    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-istio-data\") pod \"57796b75-644c-40e6-9773-d143da5fb821\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") "
Oct 31 03:50:50.594026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.593933    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57796b75-644c-40e6-9773-d143da5fb821-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57796b75-644c-40e6-9773-d143da5fb821-istiod-ca-cert\") pod \"57796b75-644c-40e6-9773-d143da5fb821\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") "
Oct 31 03:50:50.594026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.594001    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-credential-socket" pod "57796b75-644c-40e6-9773-d143da5fb821" (UID: "57796b75-644c-40e6-9773-d143da5fb821"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:50.594125 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.594018    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-workload-certs" pod "57796b75-644c-40e6-9773-d143da5fb821" (UID: "57796b75-644c-40e6-9773-d143da5fb821"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:50.594125 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.594055    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-workload-socket\") pod \"57796b75-644c-40e6-9773-d143da5fb821\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") "
Oct 31 03:50:50.594218 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.594123    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-istio-data" pod "57796b75-644c-40e6-9773-d143da5fb821" (UID: "57796b75-644c-40e6-9773-d143da5fb821"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:50.594218 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.594139    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57796b75-644c-40e6-9773-d143da5fb821-istio-token\" (UniqueName: \"kubernetes.io/projected/57796b75-644c-40e6-9773-d143da5fb821-istio-token\") pod \"57796b75-644c-40e6-9773-d143da5fb821\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") "
Oct 31 03:50:50.594218 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.594182    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57796b75-644c-40e6-9773-d143da5fb821-kube-api-access-8w5nv\" (UniqueName: \"kubernetes.io/projected/57796b75-644c-40e6-9773-d143da5fb821-kube-api-access-8w5nv\") pod \"57796b75-644c-40e6-9773-d143da5fb821\" (UID: \"57796b75-644c-40e6-9773-d143da5fb821\") "
Oct 31 03:50:50.594366 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.594278    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-workload-socket" pod "57796b75-644c-40e6-9773-d143da5fb821" (UID: "57796b75-644c-40e6-9773-d143da5fb821"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:50.594366 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.594344    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57796b75-644c-40e6-9773-d143da5fb821-istiod-ca-cert" pod "57796b75-644c-40e6-9773-d143da5fb821" (UID: "57796b75-644c-40e6-9773-d143da5fb821"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:50:50.594837 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.594762    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:50.594837 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.594788    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57796b75-644c-40e6-9773-d143da5fb821-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:50.594837 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.594804    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:50.594837 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.594817    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:50.594837 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.594836    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:50.596082 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.596056    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57796b75-644c-40e6-9773-d143da5fb821-istio-podinfo" pod "57796b75-644c-40e6-9773-d143da5fb821" (UID: "57796b75-644c-40e6-9773-d143da5fb821"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:50:50.596215 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.596141    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-istio-envoy" pod "57796b75-644c-40e6-9773-d143da5fb821" (UID: "57796b75-644c-40e6-9773-d143da5fb821"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:50:50.596371 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.596354    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57796b75-644c-40e6-9773-d143da5fb821-kube-api-access-8w5nv" pod "57796b75-644c-40e6-9773-d143da5fb821" (UID: "57796b75-644c-40e6-9773-d143da5fb821"). InnerVolumeSpecName "kube-api-access-8w5nv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:50.596454 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.596418    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57796b75-644c-40e6-9773-d143da5fb821-istio-token" pod "57796b75-644c-40e6-9773-d143da5fb821" (UID: "57796b75-644c-40e6-9773-d143da5fb821"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:50.696736 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.696647    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57796b75-644c-40e6-9773-d143da5fb821-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:50.696736 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.696685    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57796b75-644c-40e6-9773-d143da5fb821-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:50.696736 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.696703    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w5nv\" (UniqueName: \"kubernetes.io/projected/57796b75-644c-40e6-9773-d143da5fb821-kube-api-access-8w5nv\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:50.696736 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:50.696720    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57796b75-644c-40e6-9773-d143da5fb821-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:51.492164 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:51.492133    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"
Oct 31 03:50:51.710691 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:51.710654    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"]
Oct 31 03:50:51.714665 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:51.714636    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-2dhc8"]
Oct 31 03:50:52.610234 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:52.610203    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57796b75-644c-40e6-9773-d143da5fb821" path="/var/lib/kubelet/pods/57796b75-644c-40e6-9773-d143da5fb821/volumes"
Oct 31 03:50:52.680435 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:52.680402    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"]
Oct 31 03:50:52.680633 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:52.680609    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk" podUID="b49240af-df66-44be-97a9-8576b042437a" containerName="maas-api" containerID="cri-o://c2c709ea655d8d4112be47041ca6b34da98dd2c9b9082c2d10a5bdb12e6acc66" gracePeriod=30
Oct 31 03:50:53.019547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:53.019520    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"
Oct 31 03:50:53.045977 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:53.045945    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49240af-df66-44be-97a9-8576b042437a-kube-api-access-9xh9k\" (UniqueName: \"kubernetes.io/projected/b49240af-df66-44be-97a9-8576b042437a-kube-api-access-9xh9k\") pod \"b49240af-df66-44be-97a9-8576b042437a\" (UID: \"b49240af-df66-44be-97a9-8576b042437a\") "
Oct 31 03:50:53.046128 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:53.046040    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b49240af-df66-44be-97a9-8576b042437a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49240af-df66-44be-97a9-8576b042437a-openshift-service-ca\") pod \"b49240af-df66-44be-97a9-8576b042437a\" (UID: \"b49240af-df66-44be-97a9-8576b042437a\") "
Oct 31 03:50:53.046225 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:53.046208    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b49240af-df66-44be-97a9-8576b042437a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49240af-df66-44be-97a9-8576b042437a-metrics-certs\") pod \"b49240af-df66-44be-97a9-8576b042437a\" (UID: \"b49240af-df66-44be-97a9-8576b042437a\") "
Oct 31 03:50:53.046370 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:53.046355    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b49240af-df66-44be-97a9-8576b042437a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b49240af-df66-44be-97a9-8576b042437a-maas-api-tls\") pod \"b49240af-df66-44be-97a9-8576b042437a\" (UID: \"b49240af-df66-44be-97a9-8576b042437a\") "
Oct 31 03:50:53.046447 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:53.046378    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49240af-df66-44be-97a9-8576b042437a-openshift-service-ca" pod "b49240af-df66-44be-97a9-8576b042437a" (UID: "b49240af-df66-44be-97a9-8576b042437a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:50:53.046966 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:53.046879    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49240af-df66-44be-97a9-8576b042437a-openshift-service-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:53.048354 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:53.048326    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b49240af-df66-44be-97a9-8576b042437a-metrics-certs" pod "b49240af-df66-44be-97a9-8576b042437a" (UID: "b49240af-df66-44be-97a9-8576b042437a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:50:53.048452 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:53.048348    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49240af-df66-44be-97a9-8576b042437a-kube-api-access-9xh9k" pod "b49240af-df66-44be-97a9-8576b042437a" (UID: "b49240af-df66-44be-97a9-8576b042437a"). InnerVolumeSpecName "kube-api-access-9xh9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:53.048452 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:53.048361    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b49240af-df66-44be-97a9-8576b042437a-maas-api-tls" pod "b49240af-df66-44be-97a9-8576b042437a" (UID: "b49240af-df66-44be-97a9-8576b042437a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:50:53.150639 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:53.149097    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b49240af-df66-44be-97a9-8576b042437a-metrics-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:53.150639 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:53.149134    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b49240af-df66-44be-97a9-8576b042437a-maas-api-tls\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:53.150639 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:53.149152    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xh9k\" (UniqueName: \"kubernetes.io/projected/b49240af-df66-44be-97a9-8576b042437a-kube-api-access-9xh9k\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:50:53.508324 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:53.508291    4447 generic.go:356] "Generic (PLEG): container finished" podID="b49240af-df66-44be-97a9-8576b042437a" containerID="c2c709ea655d8d4112be47041ca6b34da98dd2c9b9082c2d10a5bdb12e6acc66" exitCode=0
Oct 31 03:50:53.508490 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:53.508324    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk" event={"ID":"b49240af-df66-44be-97a9-8576b042437a","Type":"ContainerDied","Data":"c2c709ea655d8d4112be47041ca6b34da98dd2c9b9082c2d10a5bdb12e6acc66"}
Oct 31 03:50:53.508490 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:53.508380    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk" event={"ID":"b49240af-df66-44be-97a9-8576b042437a","Type":"ContainerDied","Data":"4e3964823719c0b1edb8ffd0aeac74d8ca009ff8b60350363008f0574f803c54"}
Oct 31 03:50:53.508490 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:53.508404    4447 scope.go:122] "RemoveContainer" containerID="c2c709ea655d8d4112be47041ca6b34da98dd2c9b9082c2d10a5bdb12e6acc66"
Oct 31 03:50:53.508490 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:53.508336    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"
Oct 31 03:50:53.519910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:53.519867    4447 scope.go:122] "RemoveContainer" containerID="c2c709ea655d8d4112be47041ca6b34da98dd2c9b9082c2d10a5bdb12e6acc66"
Oct 31 03:50:53.520197 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:50:53.520175    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c709ea655d8d4112be47041ca6b34da98dd2c9b9082c2d10a5bdb12e6acc66\": container with ID starting with c2c709ea655d8d4112be47041ca6b34da98dd2c9b9082c2d10a5bdb12e6acc66 not found: ID does not exist" containerID="c2c709ea655d8d4112be47041ca6b34da98dd2c9b9082c2d10a5bdb12e6acc66"
Oct 31 03:50:53.520302 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:53.520203    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c709ea655d8d4112be47041ca6b34da98dd2c9b9082c2d10a5bdb12e6acc66"} err="failed to get container status \"c2c709ea655d8d4112be47041ca6b34da98dd2c9b9082c2d10a5bdb12e6acc66\": rpc error: code = NotFound desc = could not find container \"c2c709ea655d8d4112be47041ca6b34da98dd2c9b9082c2d10a5bdb12e6acc66\": container with ID starting with c2c709ea655d8d4112be47041ca6b34da98dd2c9b9082c2d10a5bdb12e6acc66 not found: ID does not exist"
Oct 31 03:50:53.712908 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:53.697203    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"]
Oct 31 03:50:53.731325 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:53.725446    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e38e21-594cb97495-nsblk"]
Oct 31 03:50:54.611443 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:50:54.611412    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b49240af-df66-44be-97a9-8576b042437a" path="/var/lib/kubelet/pods/b49240af-df66-44be-97a9-8576b042437a/volumes"
Oct 31 03:51:03.605282 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.605245    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp"]
Oct 31 03:51:03.606103 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.606085    4447 cpu_manager.go:414] "containerMap: removing container" podUID="2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc" containerName="payload-processing"
Oct 31 03:51:03.606103 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.606102    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc" containerName="payload-processing"
Oct 31 03:51:03.606296 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.606133    4447 cpu_manager.go:414] "containerMap: removing container" podUID="8f6f0c1f-5401-4d66-abbf-1cc388649280" containerName="payload-pre-processing"
Oct 31 03:51:03.606296 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.606140    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f6f0c1f-5401-4d66-abbf-1cc388649280" containerName="payload-pre-processing"
Oct 31 03:51:03.606296 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.606157    4447 cpu_manager.go:414] "containerMap: removing container" podUID="57796b75-644c-40e6-9773-d143da5fb821" containerName="istio-proxy"
Oct 31 03:51:03.606296 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.606165    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57796b75-644c-40e6-9773-d143da5fb821" containerName="istio-proxy"
Oct 31 03:51:03.606296 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.606194    4447 cpu_manager.go:414] "containerMap: removing container" podUID="b49240af-df66-44be-97a9-8576b042437a" containerName="maas-api"
Oct 31 03:51:03.606296 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.606201    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49240af-df66-44be-97a9-8576b042437a" containerName="maas-api"
Oct 31 03:51:03.606573 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.606369    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49240af-df66-44be-97a9-8576b042437a" containerName="maas-api"
Oct 31 03:51:03.606573 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.606386    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e4d1b66-e0c6-411e-9d68-59e9a1ca44dc" containerName="payload-processing"
Oct 31 03:51:03.606573 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.606394    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f6f0c1f-5401-4d66-abbf-1cc388649280" containerName="payload-pre-processing"
Oct 31 03:51:03.606573 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.606402    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="57796b75-644c-40e6-9773-d143da5fb821" containerName="istio-proxy"
Oct 31 03:51:03.610690 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.610670    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp"
Oct 31 03:51:03.667027 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.666995    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp"]
Oct 31 03:51:03.711702 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.711673    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:51:03.711702 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.711696    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 03:51:03.733306 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.733270    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d64cd46-dde8-4aae-ad8d-c11118d848e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp\" (UID: \"1d64cd46-dde8-4aae-ad8d-c11118d848e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp"
Oct 31 03:51:03.733476 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.733443    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d64cd46-dde8-4aae-ad8d-c11118d848e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp\" (UID: \"1d64cd46-dde8-4aae-ad8d-c11118d848e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp"
Oct 31 03:51:03.733514 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.733496    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44lgx\" (UniqueName: \"kubernetes.io/projected/1d64cd46-dde8-4aae-ad8d-c11118d848e6-kube-api-access-44lgx\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp\" (UID: \"1d64cd46-dde8-4aae-ad8d-c11118d848e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp"
Oct 31 03:51:03.834693 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.834657    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d64cd46-dde8-4aae-ad8d-c11118d848e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp\" (UID: \"1d64cd46-dde8-4aae-ad8d-c11118d848e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp"
Oct 31 03:51:03.834914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.834721    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44lgx\" (UniqueName: \"kubernetes.io/projected/1d64cd46-dde8-4aae-ad8d-c11118d848e6-kube-api-access-44lgx\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp\" (UID: \"1d64cd46-dde8-4aae-ad8d-c11118d848e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp"
Oct 31 03:51:03.834998 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.834945    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d64cd46-dde8-4aae-ad8d-c11118d848e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp\" (UID: \"1d64cd46-dde8-4aae-ad8d-c11118d848e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp"
Oct 31 03:51:03.835580 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.835516    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d64cd46-dde8-4aae-ad8d-c11118d848e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp\" (UID: \"1d64cd46-dde8-4aae-ad8d-c11118d848e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp"
Oct 31 03:51:03.837797 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.837776    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d64cd46-dde8-4aae-ad8d-c11118d848e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp\" (UID: \"1d64cd46-dde8-4aae-ad8d-c11118d848e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp"
Oct 31 03:51:03.896408 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.896333    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44lgx\" (UniqueName: \"kubernetes.io/projected/1d64cd46-dde8-4aae-ad8d-c11118d848e6-kube-api-access-44lgx\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp\" (UID: \"1d64cd46-dde8-4aae-ad8d-c11118d848e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp"
Oct 31 03:51:03.925821 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:03.925787    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp"
Oct 31 03:51:04.308350 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:04.308312    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp"]
Oct 31 03:51:04.309461 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:51:04.309429    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d64cd46_dde8_4aae_ad8d_c11118d848e6.slice/crio-81eb788e8f9bb6ce849c5ac4dc20c1c4dd2be95ec3d9c86b6d8cedce3b2a1d9c WatchSource:0}: Error finding container 81eb788e8f9bb6ce849c5ac4dc20c1c4dd2be95ec3d9c86b6d8cedce3b2a1d9c: Status 404 returned error can't find the container with id 81eb788e8f9bb6ce849c5ac4dc20c1c4dd2be95ec3d9c86b6d8cedce3b2a1d9c
Oct 31 03:51:04.599086 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:04.599004    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp" event={"ID":"1d64cd46-dde8-4aae-ad8d-c11118d848e6","Type":"ContainerStarted","Data":"81eb788e8f9bb6ce849c5ac4dc20c1c4dd2be95ec3d9c86b6d8cedce3b2a1d9c"}
Oct 31 03:51:05.610250 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:05.610218    4447 generic.go:356] "Generic (PLEG): container finished" podID="1d64cd46-dde8-4aae-ad8d-c11118d848e6" containerID="e01834ac92a67c31a52a687ce79fa769f05c74d7045a1e4a788265c4d09759f5" exitCode=0
Oct 31 03:51:05.610743 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:05.610313    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp" event={"ID":"1d64cd46-dde8-4aae-ad8d-c11118d848e6","Type":"ContainerDied","Data":"e01834ac92a67c31a52a687ce79fa769f05c74d7045a1e4a788265c4d09759f5"}
Oct 31 03:51:06.784106 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:06.784066    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp"
Oct 31 03:51:06.873364 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:06.873270    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d64cd46-dde8-4aae-ad8d-c11118d848e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d64cd46-dde8-4aae-ad8d-c11118d848e6-maas-api-xks-ca\") pod \"1d64cd46-dde8-4aae-ad8d-c11118d848e6\" (UID: \"1d64cd46-dde8-4aae-ad8d-c11118d848e6\") "
Oct 31 03:51:06.873528 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:06.873375    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d64cd46-dde8-4aae-ad8d-c11118d848e6-kube-api-access-44lgx\" (UniqueName: \"kubernetes.io/projected/1d64cd46-dde8-4aae-ad8d-c11118d848e6-kube-api-access-44lgx\") pod \"1d64cd46-dde8-4aae-ad8d-c11118d848e6\" (UID: \"1d64cd46-dde8-4aae-ad8d-c11118d848e6\") "
Oct 31 03:51:06.873528 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:06.873482    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d64cd46-dde8-4aae-ad8d-c11118d848e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d64cd46-dde8-4aae-ad8d-c11118d848e6-maas-api-openshift-ca\") pod \"1d64cd46-dde8-4aae-ad8d-c11118d848e6\" (UID: \"1d64cd46-dde8-4aae-ad8d-c11118d848e6\") "
Oct 31 03:51:06.873815 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:06.873788    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d64cd46-dde8-4aae-ad8d-c11118d848e6-maas-api-openshift-ca" pod "1d64cd46-dde8-4aae-ad8d-c11118d848e6" (UID: "1d64cd46-dde8-4aae-ad8d-c11118d848e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:51:06.873978 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:06.873960    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d64cd46-dde8-4aae-ad8d-c11118d848e6-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:06.875628 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:06.875600    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d64cd46-dde8-4aae-ad8d-c11118d848e6-maas-api-xks-ca" pod "1d64cd46-dde8-4aae-ad8d-c11118d848e6" (UID: "1d64cd46-dde8-4aae-ad8d-c11118d848e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:51:06.875723 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:06.875679    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d64cd46-dde8-4aae-ad8d-c11118d848e6-kube-api-access-44lgx" pod "1d64cd46-dde8-4aae-ad8d-c11118d848e6" (UID: "1d64cd46-dde8-4aae-ad8d-c11118d848e6"). InnerVolumeSpecName "kube-api-access-44lgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:51:06.975056 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:06.975023    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d64cd46-dde8-4aae-ad8d-c11118d848e6-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:06.975056 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:06.975055    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44lgx\" (UniqueName: \"kubernetes.io/projected/1d64cd46-dde8-4aae-ad8d-c11118d848e6-kube-api-access-44lgx\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:07.627318 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:07.627275    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp" event={"ID":"1d64cd46-dde8-4aae-ad8d-c11118d848e6","Type":"ContainerDied","Data":"81eb788e8f9bb6ce849c5ac4dc20c1c4dd2be95ec3d9c86b6d8cedce3b2a1d9c"}
Oct 31 03:51:07.627644 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:07.627347    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81eb788e8f9bb6ce849c5ac4dc20c1c4dd2be95ec3d9c86b6d8cedce3b2a1d9c"
Oct 31 03:51:07.627644 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:07.627288    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp"
Oct 31 03:51:15.897525 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:15.897485    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76"]
Oct 31 03:51:16.008992 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:16.008832    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd"]
Oct 31 03:51:19.003147 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:51:19.003114    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:51:20.898045 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:20.897993    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76" podUID="e0567603-ad2d-47c1-af25-d15f5a02bec2" containerName="payload-processing" containerID="cri-o://7df9d1af4da175890bcec704d80bf808072c944c1f5497acf0bf54b2db8ed1c2" gracePeriod=25
Oct 31 03:51:21.009712 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.009669    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd" podUID="6a2c31a7-238b-40e1-8b37-2aa7e59949c9" containerName="payload-pre-processing" containerID="cri-o://c9166d57774b9cc459bb62d6544eb207a17624f33972b7a516fdbd396aa5f4b3" gracePeriod=25
Oct 31 03:51:21.190299 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.189864    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76"
Oct 31 03:51:21.259972 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.259936    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0567603-ad2d-47c1-af25-d15f5a02bec2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0567603-ad2d-47c1-af25-d15f5a02bec2-plugins-config-volume\") pod \"e0567603-ad2d-47c1-af25-d15f5a02bec2\" (UID: \"e0567603-ad2d-47c1-af25-d15f5a02bec2\") "
Oct 31 03:51:21.260166 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.260045    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0567603-ad2d-47c1-af25-d15f5a02bec2-kube-api-access-ntrct\" (UniqueName: \"kubernetes.io/projected/e0567603-ad2d-47c1-af25-d15f5a02bec2-kube-api-access-ntrct\") pod \"e0567603-ad2d-47c1-af25-d15f5a02bec2\" (UID: \"e0567603-ad2d-47c1-af25-d15f5a02bec2\") "
Oct 31 03:51:21.260356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.260327    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0567603-ad2d-47c1-af25-d15f5a02bec2-plugins-config-volume" pod "e0567603-ad2d-47c1-af25-d15f5a02bec2" (UID: "e0567603-ad2d-47c1-af25-d15f5a02bec2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:51:21.261036 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.261010    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0567603-ad2d-47c1-af25-d15f5a02bec2-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:21.265066 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.265028    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0567603-ad2d-47c1-af25-d15f5a02bec2-kube-api-access-ntrct" pod "e0567603-ad2d-47c1-af25-d15f5a02bec2" (UID: "e0567603-ad2d-47c1-af25-d15f5a02bec2"). InnerVolumeSpecName "kube-api-access-ntrct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:51:21.290216 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.290191    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd"
Oct 31 03:51:21.361520 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.361492    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a2c31a7-238b-40e1-8b37-2aa7e59949c9-kube-api-access-5w8bw\" (UniqueName: \"kubernetes.io/projected/6a2c31a7-238b-40e1-8b37-2aa7e59949c9-kube-api-access-5w8bw\") pod \"6a2c31a7-238b-40e1-8b37-2aa7e59949c9\" (UID: \"6a2c31a7-238b-40e1-8b37-2aa7e59949c9\") "
Oct 31 03:51:21.361729 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.361614    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a2c31a7-238b-40e1-8b37-2aa7e59949c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a2c31a7-238b-40e1-8b37-2aa7e59949c9-plugins-config-volume\") pod \"6a2c31a7-238b-40e1-8b37-2aa7e59949c9\" (UID: \"6a2c31a7-238b-40e1-8b37-2aa7e59949c9\") "
Oct 31 03:51:21.362106 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.362076    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a2c31a7-238b-40e1-8b37-2aa7e59949c9-plugins-config-volume" pod "6a2c31a7-238b-40e1-8b37-2aa7e59949c9" (UID: "6a2c31a7-238b-40e1-8b37-2aa7e59949c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:51:21.363589 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.363564    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntrct\" (UniqueName: \"kubernetes.io/projected/e0567603-ad2d-47c1-af25-d15f5a02bec2-kube-api-access-ntrct\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:21.363589 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.363589    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a2c31a7-238b-40e1-8b37-2aa7e59949c9-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:21.364000 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.363982    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a2c31a7-238b-40e1-8b37-2aa7e59949c9-kube-api-access-5w8bw" pod "6a2c31a7-238b-40e1-8b37-2aa7e59949c9" (UID: "6a2c31a7-238b-40e1-8b37-2aa7e59949c9"). InnerVolumeSpecName "kube-api-access-5w8bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:51:21.465014 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.464976    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w8bw\" (UniqueName: \"kubernetes.io/projected/6a2c31a7-238b-40e1-8b37-2aa7e59949c9-kube-api-access-5w8bw\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:21.764080 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.763989    4447 generic.go:356] "Generic (PLEG): container finished" podID="6a2c31a7-238b-40e1-8b37-2aa7e59949c9" containerID="c9166d57774b9cc459bb62d6544eb207a17624f33972b7a516fdbd396aa5f4b3" exitCode=0
Oct 31 03:51:21.764080 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.764069    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd"
Oct 31 03:51:21.764281 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.764071    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd" event={"ID":"6a2c31a7-238b-40e1-8b37-2aa7e59949c9","Type":"ContainerDied","Data":"c9166d57774b9cc459bb62d6544eb207a17624f33972b7a516fdbd396aa5f4b3"}
Oct 31 03:51:21.764281 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.764127    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd" event={"ID":"6a2c31a7-238b-40e1-8b37-2aa7e59949c9","Type":"ContainerDied","Data":"6eeb1448b4f09ef376c7d0871ed6ecbc2bad02291c761f0819bc8020497781b8"}
Oct 31 03:51:21.764281 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.764150    4447 scope.go:122] "RemoveContainer" containerID="c9166d57774b9cc459bb62d6544eb207a17624f33972b7a516fdbd396aa5f4b3"
Oct 31 03:51:21.765754 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.765728    4447 generic.go:356] "Generic (PLEG): container finished" podID="e0567603-ad2d-47c1-af25-d15f5a02bec2" containerID="7df9d1af4da175890bcec704d80bf808072c944c1f5497acf0bf54b2db8ed1c2" exitCode=0
Oct 31 03:51:21.765877 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.765781    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76" event={"ID":"e0567603-ad2d-47c1-af25-d15f5a02bec2","Type":"ContainerDied","Data":"7df9d1af4da175890bcec704d80bf808072c944c1f5497acf0bf54b2db8ed1c2"}
Oct 31 03:51:21.765877 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.765808    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76"
Oct 31 03:51:21.765877 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.765829    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76" event={"ID":"e0567603-ad2d-47c1-af25-d15f5a02bec2","Type":"ContainerDied","Data":"05a4444e8a9658e37bd8350044abaf02ec73b9ffd02c98a6139295ec3161b1b7"}
Oct 31 03:51:21.777376 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.777356    4447 scope.go:122] "RemoveContainer" containerID="c9166d57774b9cc459bb62d6544eb207a17624f33972b7a516fdbd396aa5f4b3"
Oct 31 03:51:21.777748 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:51:21.777725    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9166d57774b9cc459bb62d6544eb207a17624f33972b7a516fdbd396aa5f4b3\": container with ID starting with c9166d57774b9cc459bb62d6544eb207a17624f33972b7a516fdbd396aa5f4b3 not found: ID does not exist" containerID="c9166d57774b9cc459bb62d6544eb207a17624f33972b7a516fdbd396aa5f4b3"
Oct 31 03:51:21.777790 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.777758    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9166d57774b9cc459bb62d6544eb207a17624f33972b7a516fdbd396aa5f4b3"} err="failed to get container status \"c9166d57774b9cc459bb62d6544eb207a17624f33972b7a516fdbd396aa5f4b3\": rpc error: code = NotFound desc = could not find container \"c9166d57774b9cc459bb62d6544eb207a17624f33972b7a516fdbd396aa5f4b3\": container with ID starting with c9166d57774b9cc459bb62d6544eb207a17624f33972b7a516fdbd396aa5f4b3 not found: ID does not exist"
Oct 31 03:51:21.777790 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.777775    4447 scope.go:122] "RemoveContainer" containerID="7df9d1af4da175890bcec704d80bf808072c944c1f5497acf0bf54b2db8ed1c2"
Oct 31 03:51:21.788934 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.788907    4447 scope.go:122] "RemoveContainer" containerID="7df9d1af4da175890bcec704d80bf808072c944c1f5497acf0bf54b2db8ed1c2"
Oct 31 03:51:21.789270 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:51:21.789250    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7df9d1af4da175890bcec704d80bf808072c944c1f5497acf0bf54b2db8ed1c2\": container with ID starting with 7df9d1af4da175890bcec704d80bf808072c944c1f5497acf0bf54b2db8ed1c2 not found: ID does not exist" containerID="7df9d1af4da175890bcec704d80bf808072c944c1f5497acf0bf54b2db8ed1c2"
Oct 31 03:51:21.789344 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.789279    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7df9d1af4da175890bcec704d80bf808072c944c1f5497acf0bf54b2db8ed1c2"} err="failed to get container status \"7df9d1af4da175890bcec704d80bf808072c944c1f5497acf0bf54b2db8ed1c2\": rpc error: code = NotFound desc = could not find container \"7df9d1af4da175890bcec704d80bf808072c944c1f5497acf0bf54b2db8ed1c2\": container with ID starting with 7df9d1af4da175890bcec704d80bf808072c944c1f5497acf0bf54b2db8ed1c2 not found: ID does not exist"
Oct 31 03:51:21.931613 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.931579    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd"]
Oct 31 03:51:21.981304 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:21.981271    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-2accc9-8687b4c5c5-4jdbd"]
Oct 31 03:51:22.045162 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:22.045063    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76"]
Oct 31 03:51:22.049223 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:22.049200    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-2accc9-5cd5dd98dc-r7p76"]
Oct 31 03:51:22.612273 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:22.612231    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a2c31a7-238b-40e1-8b37-2aa7e59949c9" path="/var/lib/kubelet/pods/6a2c31a7-238b-40e1-8b37-2aa7e59949c9/volumes"
Oct 31 03:51:22.612814 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:22.612792    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0567603-ad2d-47c1-af25-d15f5a02bec2" path="/var/lib/kubelet/pods/e0567603-ad2d-47c1-af25-d15f5a02bec2/volumes"
Oct 31 03:51:23.557911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:23.540144    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"]
Oct 31 03:51:23.557911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:23.541068    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8" podUID="bbf932c9-3168-4564-aab2-5391dd41cc1a" containerName="maas-api" containerID="cri-o://d497fa4f32bc8b32e49d06065d13e0184679023e91f0759059e8abee44420992" gracePeriod=30
Oct 31 03:51:23.791937 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:23.790824    4447 generic.go:356] "Generic (PLEG): container finished" podID="bbf932c9-3168-4564-aab2-5391dd41cc1a" containerID="d497fa4f32bc8b32e49d06065d13e0184679023e91f0759059e8abee44420992" exitCode=0
Oct 31 03:51:23.791937 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:23.790979    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8" event={"ID":"bbf932c9-3168-4564-aab2-5391dd41cc1a","Type":"ContainerDied","Data":"d497fa4f32bc8b32e49d06065d13e0184679023e91f0759059e8abee44420992"}
Oct 31 03:51:23.985910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:23.978394    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"
Oct 31 03:51:24.071640 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:24.071610    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbf932c9-3168-4564-aab2-5391dd41cc1a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbf932c9-3168-4564-aab2-5391dd41cc1a-maas-api-tls\") pod \"bbf932c9-3168-4564-aab2-5391dd41cc1a\" (UID: \"bbf932c9-3168-4564-aab2-5391dd41cc1a\") "
Oct 31 03:51:24.071829 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:24.071815    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbf932c9-3168-4564-aab2-5391dd41cc1a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbf932c9-3168-4564-aab2-5391dd41cc1a-openshift-service-ca\") pod \"bbf932c9-3168-4564-aab2-5391dd41cc1a\" (UID: \"bbf932c9-3168-4564-aab2-5391dd41cc1a\") "
Oct 31 03:51:24.072007 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:24.071994    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf932c9-3168-4564-aab2-5391dd41cc1a-kube-api-access-mqrt6\" (UniqueName: \"kubernetes.io/projected/bbf932c9-3168-4564-aab2-5391dd41cc1a-kube-api-access-mqrt6\") pod \"bbf932c9-3168-4564-aab2-5391dd41cc1a\" (UID: \"bbf932c9-3168-4564-aab2-5391dd41cc1a\") "
Oct 31 03:51:24.072080 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:24.072019    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbf932c9-3168-4564-aab2-5391dd41cc1a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf932c9-3168-4564-aab2-5391dd41cc1a-metrics-certs\") pod \"bbf932c9-3168-4564-aab2-5391dd41cc1a\" (UID: \"bbf932c9-3168-4564-aab2-5391dd41cc1a\") "
Oct 31 03:51:24.072190 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:24.072164    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbf932c9-3168-4564-aab2-5391dd41cc1a-openshift-service-ca" pod "bbf932c9-3168-4564-aab2-5391dd41cc1a" (UID: "bbf932c9-3168-4564-aab2-5391dd41cc1a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:51:24.072981 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:24.072948    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbf932c9-3168-4564-aab2-5391dd41cc1a-openshift-service-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:24.074304 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:24.074280    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbf932c9-3168-4564-aab2-5391dd41cc1a-metrics-certs" pod "bbf932c9-3168-4564-aab2-5391dd41cc1a" (UID: "bbf932c9-3168-4564-aab2-5391dd41cc1a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:51:24.074390 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:24.074341    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbf932c9-3168-4564-aab2-5391dd41cc1a-maas-api-tls" pod "bbf932c9-3168-4564-aab2-5391dd41cc1a" (UID: "bbf932c9-3168-4564-aab2-5391dd41cc1a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:51:24.074435 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:24.074397    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf932c9-3168-4564-aab2-5391dd41cc1a-kube-api-access-mqrt6" pod "bbf932c9-3168-4564-aab2-5391dd41cc1a" (UID: "bbf932c9-3168-4564-aab2-5391dd41cc1a"). InnerVolumeSpecName "kube-api-access-mqrt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:51:24.174398 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:24.174312    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqrt6\" (UniqueName: \"kubernetes.io/projected/bbf932c9-3168-4564-aab2-5391dd41cc1a-kube-api-access-mqrt6\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:24.174398 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:24.174341    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf932c9-3168-4564-aab2-5391dd41cc1a-metrics-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:24.174398 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:24.174351    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbf932c9-3168-4564-aab2-5391dd41cc1a-maas-api-tls\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:24.801262 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:24.801236    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"
Oct 31 03:51:24.801717 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:24.801281    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8" event={"ID":"bbf932c9-3168-4564-aab2-5391dd41cc1a","Type":"ContainerDied","Data":"d1637d182f49669215488ee731abbfedc7b797afc69b5e6a46ede28ee0a92982"}
Oct 31 03:51:24.801717 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:24.801347    4447 scope.go:122] "RemoveContainer" containerID="d497fa4f32bc8b32e49d06065d13e0184679023e91f0759059e8abee44420992"
Oct 31 03:51:24.968177 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:24.968146    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"]
Oct 31 03:51:25.017985 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:25.017949    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-096812-856c94495f-zh5m8"]
Oct 31 03:51:26.611664 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:26.611626    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbf932c9-3168-4564-aab2-5391dd41cc1a" path="/var/lib/kubelet/pods/bbf932c9-3168-4564-aab2-5391dd41cc1a/volumes"
Oct 31 03:51:34.055929 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:34.055872    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"]
Oct 31 03:51:34.056643 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:34.056104    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz" podUID="80bc3946-0385-4655-a0eb-3266a188f7ab" containerName="istio-proxy" containerID="cri-o://795351b27ca0ba3a922bcf786f45a2cfa79947e03f4a28921288e4a4a90b36d7" gracePeriod=30
Oct 31 03:51:34.084862 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:34.084832    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"]
Oct 31 03:51:34.085061 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:34.085041    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6" podUID="5245fec2-9af5-4816-87de-e1d9a78930b2" containerName="istio-proxy" containerID="cri-o://c07341a7a867c0bd3a6c0c4b635124ff8c096a1acc4f451e20ce43bf34b1feea" gracePeriod=30
Oct 31 03:51:36.336316 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.336285    4447 scope.go:122] "RemoveContainer" containerID="18ed63cae172bd167174279633990c21a6512ee34c0c714cd21da00ebdcb2e65"
Oct 31 03:51:36.661524 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.661437    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx"]
Oct 31 03:51:36.662228 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.662207    4447 cpu_manager.go:414] "containerMap: removing container" podUID="6a2c31a7-238b-40e1-8b37-2aa7e59949c9" containerName="payload-pre-processing"
Oct 31 03:51:36.662228 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.662225    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a2c31a7-238b-40e1-8b37-2aa7e59949c9" containerName="payload-pre-processing"
Oct 31 03:51:36.662430 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.662244    4447 cpu_manager.go:414] "containerMap: removing container" podUID="e0567603-ad2d-47c1-af25-d15f5a02bec2" containerName="payload-processing"
Oct 31 03:51:36.662430 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.662249    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0567603-ad2d-47c1-af25-d15f5a02bec2" containerName="payload-processing"
Oct 31 03:51:36.662430 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.662285    4447 cpu_manager.go:414] "containerMap: removing container" podUID="bbf932c9-3168-4564-aab2-5391dd41cc1a" containerName="maas-api"
Oct 31 03:51:36.662430 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.662290    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbf932c9-3168-4564-aab2-5391dd41cc1a" containerName="maas-api"
Oct 31 03:51:36.662430 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.662297    4447 cpu_manager.go:414] "containerMap: removing container" podUID="1d64cd46-dde8-4aae-ad8d-c11118d848e6" containerName="revoke-keys"
Oct 31 03:51:36.662430 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.662301    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d64cd46-dde8-4aae-ad8d-c11118d848e6" containerName="revoke-keys"
Oct 31 03:51:36.662430 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.662403    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a2c31a7-238b-40e1-8b37-2aa7e59949c9" containerName="payload-pre-processing"
Oct 31 03:51:36.662430 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.662412    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0567603-ad2d-47c1-af25-d15f5a02bec2" containerName="payload-processing"
Oct 31 03:51:36.662430 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.662427    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbf932c9-3168-4564-aab2-5391dd41cc1a" containerName="maas-api"
Oct 31 03:51:36.662430 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.662434    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d64cd46-dde8-4aae-ad8d-c11118d848e6" containerName="revoke-keys"
Oct 31 03:51:36.667966 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.667941    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx"
Oct 31 03:51:36.720532 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.720500    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e02dd5\""
Oct 31 03:51:36.723864 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.723834    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx"]
Oct 31 03:51:36.787139 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.769363    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e02dd5-dockercfg-jmg8r\""
Oct 31 03:51:36.819916 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.817598    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afd09e9-4640-4fb7-87a3-1cc053929b2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx\" (UID: \"6afd09e9-4640-4fb7-87a3-1cc053929b2c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx"
Oct 31 03:51:36.819916 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.817684    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hggv5\" (UniqueName: \"kubernetes.io/projected/6afd09e9-4640-4fb7-87a3-1cc053929b2c-kube-api-access-hggv5\") pod \"payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx\" (UID: \"6afd09e9-4640-4fb7-87a3-1cc053929b2c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx"
Oct 31 03:51:36.861500 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.857932    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn"]
Oct 31 03:51:36.863995 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.863969    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn"
Oct 31 03:51:36.893232 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.893202    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn"]
Oct 31 03:51:36.919027 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.918928    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5f34fa-2f2e-4fe0-b77f-02667175ac6a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn\" (UID: \"8b5f34fa-2f2e-4fe0-b77f-02667175ac6a\") " pod="openshift-ingress/payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn"
Oct 31 03:51:36.919027 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.919028    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sclkf\" (UniqueName: \"kubernetes.io/projected/8b5f34fa-2f2e-4fe0-b77f-02667175ac6a-kube-api-access-sclkf\") pod \"payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn\" (UID: \"8b5f34fa-2f2e-4fe0-b77f-02667175ac6a\") " pod="openshift-ingress/payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn"
Oct 31 03:51:36.919317 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.919205    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afd09e9-4640-4fb7-87a3-1cc053929b2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx\" (UID: \"6afd09e9-4640-4fb7-87a3-1cc053929b2c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx"
Oct 31 03:51:36.919317 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.919286    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hggv5\" (UniqueName: \"kubernetes.io/projected/6afd09e9-4640-4fb7-87a3-1cc053929b2c-kube-api-access-hggv5\") pod \"payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx\" (UID: \"6afd09e9-4640-4fb7-87a3-1cc053929b2c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx"
Oct 31 03:51:36.919871 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.919850    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afd09e9-4640-4fb7-87a3-1cc053929b2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx\" (UID: \"6afd09e9-4640-4fb7-87a3-1cc053929b2c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx"
Oct 31 03:51:36.982266 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.982236    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hggv5\" (UniqueName: \"kubernetes.io/projected/6afd09e9-4640-4fb7-87a3-1cc053929b2c-kube-api-access-hggv5\") pod \"payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx\" (UID: \"6afd09e9-4640-4fb7-87a3-1cc053929b2c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx"
Oct 31 03:51:36.985324 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:36.985289    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx"
Oct 31 03:51:37.020502 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:37.020464    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5f34fa-2f2e-4fe0-b77f-02667175ac6a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn\" (UID: \"8b5f34fa-2f2e-4fe0-b77f-02667175ac6a\") " pod="openshift-ingress/payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn"
Oct 31 03:51:37.020668 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:37.020585    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sclkf\" (UniqueName: \"kubernetes.io/projected/8b5f34fa-2f2e-4fe0-b77f-02667175ac6a-kube-api-access-sclkf\") pod \"payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn\" (UID: \"8b5f34fa-2f2e-4fe0-b77f-02667175ac6a\") " pod="openshift-ingress/payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn"
Oct 31 03:51:37.021414 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:37.021390    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5f34fa-2f2e-4fe0-b77f-02667175ac6a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn\" (UID: \"8b5f34fa-2f2e-4fe0-b77f-02667175ac6a\") " pod="openshift-ingress/payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn"
Oct 31 03:51:37.083107 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:37.083075    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sclkf\" (UniqueName: \"kubernetes.io/projected/8b5f34fa-2f2e-4fe0-b77f-02667175ac6a-kube-api-access-sclkf\") pod \"payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn\" (UID: \"8b5f34fa-2f2e-4fe0-b77f-02667175ac6a\") " pod="openshift-ingress/payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn"
Oct 31 03:51:37.185168 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:37.185085    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn"
Oct 31 03:51:37.368540 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:37.368512    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx"]
Oct 31 03:51:37.368923 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:51:37.368900    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6afd09e9_4640_4fb7_87a3_1cc053929b2c.slice/crio-5ff9dbbcddf7b518cc25b21c20c7ccfc7d898442a8ef9b70074fee37a07c71ea WatchSource:0}: Error finding container 5ff9dbbcddf7b518cc25b21c20c7ccfc7d898442a8ef9b70074fee37a07c71ea: Status 404 returned error can't find the container with id 5ff9dbbcddf7b518cc25b21c20c7ccfc7d898442a8ef9b70074fee37a07c71ea
Oct 31 03:51:37.551855 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:37.551823    4447 patch_prober.go:29] interesting pod/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:51:37.552074 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:37.551906    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz" podUID="80bc3946-0385-4655-a0eb-3266a188f7ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:51:37.567863 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:37.567832    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn"]
Oct 31 03:51:37.568235 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:51:37.568211    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5f34fa_2f2e_4fe0_b77f_02667175ac6a.slice/crio-f91b5da5bf7e7b5c1f96846bf3a5fdb8771e79fb958bbcc218c1e609367ee00b WatchSource:0}: Error finding container f91b5da5bf7e7b5c1f96846bf3a5fdb8771e79fb958bbcc218c1e609367ee00b: Status 404 returned error can't find the container with id f91b5da5bf7e7b5c1f96846bf3a5fdb8771e79fb958bbcc218c1e609367ee00b
Oct 31 03:51:37.948850 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:37.948806    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn" event={"ID":"8b5f34fa-2f2e-4fe0-b77f-02667175ac6a","Type":"ContainerStarted","Data":"6ec7952c37e52cd280e3438bed64d3ef1e1eb215cd1bb25df091e473d495019d"}
Oct 31 03:51:37.949084 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:37.948877    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn"
Oct 31 03:51:37.949084 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:37.948920    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn" event={"ID":"8b5f34fa-2f2e-4fe0-b77f-02667175ac6a","Type":"ContainerStarted","Data":"f91b5da5bf7e7b5c1f96846bf3a5fdb8771e79fb958bbcc218c1e609367ee00b"}
Oct 31 03:51:37.950625 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:37.950600    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx" event={"ID":"6afd09e9-4640-4fb7-87a3-1cc053929b2c","Type":"ContainerStarted","Data":"c15e5507bbbe11a8601252b30f00e8e81777356cc8d8102b57224602a8fff97a"}
Oct 31 03:51:37.950755 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:37.950651    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx" event={"ID":"6afd09e9-4640-4fb7-87a3-1cc053929b2c","Type":"ContainerStarted","Data":"5ff9dbbcddf7b518cc25b21c20c7ccfc7d898442a8ef9b70074fee37a07c71ea"}
Oct 31 03:51:37.950755 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:37.950691    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx"
Oct 31 03:51:38.070677 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.070611    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn" podStartSLOduration=2.070591759 podStartE2EDuration="2.070591759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:51:38.06541604 +0000 UTC m=+1923.576398436" watchObservedRunningTime="2026-10-31 03:51:38.070591759 +0000 UTC m=+1923.581574138"
Oct 31 03:51:38.493273 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.493223    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx" podStartSLOduration=2.493206851 podStartE2EDuration="2.493206851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:51:38.205367544 +0000 UTC m=+1923.716349921" watchObservedRunningTime="2026-10-31 03:51:38.493206851 +0000 UTC m=+1924.004189228"
Oct 31 03:51:38.493741 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.493727    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"]
Oct 31 03:51:38.499059 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.499040    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.544810 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.544778    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-envoy\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.544985 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.544853    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-podinfo\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.544985 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.544934    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-credential-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.544985 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.544982    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-data\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.545109 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.545011    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-workload-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.545109 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.545088    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istiod-ca-cert\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.545169 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.545128    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-workload-certs\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.545288 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.545274    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-token\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.545330 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.545315    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5g8q\" (UniqueName: \"kubernetes.io/projected/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-kube-api-access-n5g8q\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.551156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.551136    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-1783b3-openshift-default-dockercfg-2fskz\""
Oct 31 03:51:38.621669 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.621633    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"]
Oct 31 03:51:38.646743 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.646708    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-data\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.646945 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.646777    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-workload-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.646945 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.646701    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-data\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.647091 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.647027    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istiod-ca-cert\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.647159 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.647133    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-workload-certs\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.647321 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.647301    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-token\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.647399 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.647372    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5g8q\" (UniqueName: \"kubernetes.io/projected/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-kube-api-access-n5g8q\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.647461 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.647400    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-workload-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.647517 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.647459    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-envoy\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.647592 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.647571    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-podinfo\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.647650 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.647612    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-credential-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.648560 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.648533    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-credential-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.648875 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.648851    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-workload-certs\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.649031 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.649016    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istiod-ca-cert\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.651715 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.651689    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-envoy\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.652496 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.652233    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-podinfo\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.709767 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.709731    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-token\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.709952 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.709795    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5g8q\" (UniqueName: \"kubernetes.io/projected/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-kube-api-access-n5g8q\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:38.813927 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:38.813805    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:39.250296 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:51:39.250266    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd66aaca_cbe4_414b_a056_551ed0c3f0b6.slice/crio-bb1ef9c9ecbb3dfc298a0aa31bf7345beddbd6fdb7ec0631e405edf2a91aaafe WatchSource:0}: Error finding container bb1ef9c9ecbb3dfc298a0aa31bf7345beddbd6fdb7ec0631e405edf2a91aaafe: Status 404 returned error can't find the container with id bb1ef9c9ecbb3dfc298a0aa31bf7345beddbd6fdb7ec0631e405edf2a91aaafe
Oct 31 03:51:39.250809 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.250549    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"]
Oct 31 03:51:39.252844 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.252806    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:51:39.252948 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.252909    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:51:39.252992 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.252955    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:51:39.389993 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.389965    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:51:39.406279 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.406254    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:51:39.458585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.457747    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-envoy\") pod \"80bc3946-0385-4655-a0eb-3266a188f7ab\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") "
Oct 31 03:51:39.458585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.457817    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-data\") pod \"5245fec2-9af5-4816-87de-e1d9a78930b2\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") "
Oct 31 03:51:39.458585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.457967    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-token\" (UniqueName: \"kubernetes.io/projected/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-token\") pod \"80bc3946-0385-4655-a0eb-3266a188f7ab\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") "
Oct 31 03:51:39.458585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.458016    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-podinfo\") pod \"80bc3946-0385-4655-a0eb-3266a188f7ab\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") "
Oct 31 03:51:39.458585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.458059    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-workload-certs\") pod \"80bc3946-0385-4655-a0eb-3266a188f7ab\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") "
Oct 31 03:51:39.458585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.458111    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-workload-socket\") pod \"80bc3946-0385-4655-a0eb-3266a188f7ab\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") "
Oct 31 03:51:39.458585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.458160    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5245fec2-9af5-4816-87de-e1d9a78930b2-kube-api-access-lrr85\" (UniqueName: \"kubernetes.io/projected/5245fec2-9af5-4816-87de-e1d9a78930b2-kube-api-access-lrr85\") pod \"5245fec2-9af5-4816-87de-e1d9a78930b2\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") "
Oct 31 03:51:39.458585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.458183    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-workload-socket\") pod \"5245fec2-9af5-4816-87de-e1d9a78930b2\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") "
Oct 31 03:51:39.458585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.458199    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-data" pod "5245fec2-9af5-4816-87de-e1d9a78930b2" (UID: "5245fec2-9af5-4816-87de-e1d9a78930b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:51:39.458585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.458236    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-token\" (UniqueName: \"kubernetes.io/projected/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-token\") pod \"5245fec2-9af5-4816-87de-e1d9a78930b2\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") "
Oct 31 03:51:39.458585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.458271    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-envoy\") pod \"5245fec2-9af5-4816-87de-e1d9a78930b2\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") "
Oct 31 03:51:39.458585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.458333    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80bc3946-0385-4655-a0eb-3266a188f7ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80bc3946-0385-4655-a0eb-3266a188f7ab-istiod-ca-cert\") pod \"80bc3946-0385-4655-a0eb-3266a188f7ab\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") "
Oct 31 03:51:39.458585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.458361    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-data\") pod \"80bc3946-0385-4655-a0eb-3266a188f7ab\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") "
Oct 31 03:51:39.458585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.458421    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-workload-certs" pod "80bc3946-0385-4655-a0eb-3266a188f7ab" (UID: "80bc3946-0385-4655-a0eb-3266a188f7ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:51:39.458585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.458422    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-workload-certs\") pod \"5245fec2-9af5-4816-87de-e1d9a78930b2\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") "
Oct 31 03:51:39.458585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.458486    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-credential-socket\") pod \"5245fec2-9af5-4816-87de-e1d9a78930b2\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") "
Oct 31 03:51:39.458585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.458520    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5245fec2-9af5-4816-87de-e1d9a78930b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5245fec2-9af5-4816-87de-e1d9a78930b2-istiod-ca-cert\") pod \"5245fec2-9af5-4816-87de-e1d9a78930b2\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") "
Oct 31 03:51:39.458585 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.458589    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-credential-socket\") pod \"80bc3946-0385-4655-a0eb-3266a188f7ab\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") "
Oct 31 03:51:39.459548 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.458641    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-podinfo\") pod \"5245fec2-9af5-4816-87de-e1d9a78930b2\" (UID: \"5245fec2-9af5-4816-87de-e1d9a78930b2\") "
Oct 31 03:51:39.459548 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.458688    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80bc3946-0385-4655-a0eb-3266a188f7ab-kube-api-access-9w7n6\" (UniqueName: \"kubernetes.io/projected/80bc3946-0385-4655-a0eb-3266a188f7ab-kube-api-access-9w7n6\") pod \"80bc3946-0385-4655-a0eb-3266a188f7ab\" (UID: \"80bc3946-0385-4655-a0eb-3266a188f7ab\") "
Oct 31 03:51:39.459548 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.459478    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:39.459548 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.459499    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:39.459866 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.459832    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-workload-certs" pod "5245fec2-9af5-4816-87de-e1d9a78930b2" (UID: "5245fec2-9af5-4816-87de-e1d9a78930b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:51:39.460103 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.460084    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-credential-socket" pod "5245fec2-9af5-4816-87de-e1d9a78930b2" (UID: "5245fec2-9af5-4816-87de-e1d9a78930b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:51:39.460868 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.460846    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-workload-socket" pod "5245fec2-9af5-4816-87de-e1d9a78930b2" (UID: "5245fec2-9af5-4816-87de-e1d9a78930b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:51:39.462261 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.461187    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-workload-socket" pod "80bc3946-0385-4655-a0eb-3266a188f7ab" (UID: "80bc3946-0385-4655-a0eb-3266a188f7ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:51:39.462261 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.461739    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-data" pod "80bc3946-0385-4655-a0eb-3266a188f7ab" (UID: "80bc3946-0385-4655-a0eb-3266a188f7ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:51:39.462261 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.462012    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5245fec2-9af5-4816-87de-e1d9a78930b2-kube-api-access-lrr85" pod "5245fec2-9af5-4816-87de-e1d9a78930b2" (UID: "5245fec2-9af5-4816-87de-e1d9a78930b2"). InnerVolumeSpecName "kube-api-access-lrr85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:51:39.462261 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.462124    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80bc3946-0385-4655-a0eb-3266a188f7ab-kube-api-access-9w7n6" pod "80bc3946-0385-4655-a0eb-3266a188f7ab" (UID: "80bc3946-0385-4655-a0eb-3266a188f7ab"). InnerVolumeSpecName "kube-api-access-9w7n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:51:39.462261 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.462233    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-credential-socket" pod "80bc3946-0385-4655-a0eb-3266a188f7ab" (UID: "80bc3946-0385-4655-a0eb-3266a188f7ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:51:39.462702 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.462493    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80bc3946-0385-4655-a0eb-3266a188f7ab-istiod-ca-cert" pod "80bc3946-0385-4655-a0eb-3266a188f7ab" (UID: "80bc3946-0385-4655-a0eb-3266a188f7ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:51:39.462702 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.462525    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5245fec2-9af5-4816-87de-e1d9a78930b2-istiod-ca-cert" pod "5245fec2-9af5-4816-87de-e1d9a78930b2" (UID: "5245fec2-9af5-4816-87de-e1d9a78930b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:51:39.465485 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.465449    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-token" pod "80bc3946-0385-4655-a0eb-3266a188f7ab" (UID: "80bc3946-0385-4655-a0eb-3266a188f7ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:51:39.465590 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.465556    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-podinfo" pod "5245fec2-9af5-4816-87de-e1d9a78930b2" (UID: "5245fec2-9af5-4816-87de-e1d9a78930b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:51:39.466273 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.466246    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-token" pod "5245fec2-9af5-4816-87de-e1d9a78930b2" (UID: "5245fec2-9af5-4816-87de-e1d9a78930b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:51:39.466360 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.466286    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-envoy" pod "80bc3946-0385-4655-a0eb-3266a188f7ab" (UID: "80bc3946-0385-4655-a0eb-3266a188f7ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:51:39.466426 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.466400    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-podinfo" pod "80bc3946-0385-4655-a0eb-3266a188f7ab" (UID: "80bc3946-0385-4655-a0eb-3266a188f7ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:51:39.468199 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.468175    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-envoy" pod "5245fec2-9af5-4816-87de-e1d9a78930b2" (UID: "5245fec2-9af5-4816-87de-e1d9a78930b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:51:39.560930 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.560877    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:39.560930 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.560931    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:39.561483 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.560946    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:39.561483 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.560960    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrr85\" (UniqueName: \"kubernetes.io/projected/5245fec2-9af5-4816-87de-e1d9a78930b2-kube-api-access-lrr85\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:39.561483 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.560975    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:39.561483 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.560990    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:39.561483 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.561004    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:39.561483 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.561017    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80bc3946-0385-4655-a0eb-3266a188f7ab-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:39.561483 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.561030    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:39.561483 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.561042    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:39.561483 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.561055    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5245fec2-9af5-4816-87de-e1d9a78930b2-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:39.561483 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.561068    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5245fec2-9af5-4816-87de-e1d9a78930b2-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:39.561483 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.561081    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:39.561483 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.561094    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5245fec2-9af5-4816-87de-e1d9a78930b2-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:39.561483 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.561107    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w7n6\" (UniqueName: \"kubernetes.io/projected/80bc3946-0385-4655-a0eb-3266a188f7ab-kube-api-access-9w7n6\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:39.561483 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.561121    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80bc3946-0385-4655-a0eb-3266a188f7ab-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:39.970560 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.970524    4447 generic.go:356] "Generic (PLEG): container finished" podID="5245fec2-9af5-4816-87de-e1d9a78930b2" containerID="c07341a7a867c0bd3a6c0c4b635124ff8c096a1acc4f451e20ce43bf34b1feea" exitCode=0
Oct 31 03:51:39.970725 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.970659    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6" event={"ID":"5245fec2-9af5-4816-87de-e1d9a78930b2","Type":"ContainerDied","Data":"c07341a7a867c0bd3a6c0c4b635124ff8c096a1acc4f451e20ce43bf34b1feea"}
Oct 31 03:51:39.970725 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.970705    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6" event={"ID":"5245fec2-9af5-4816-87de-e1d9a78930b2","Type":"ContainerDied","Data":"a3207a37b0b28c32d6395f541e3e411aa041e19ec8fcd7491f3e7366732954cd"}
Oct 31 03:51:39.970792 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.970726    4447 scope.go:122] "RemoveContainer" containerID="c07341a7a867c0bd3a6c0c4b635124ff8c096a1acc4f451e20ce43bf34b1feea"
Oct 31 03:51:39.970862 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.970851    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"
Oct 31 03:51:39.974169 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.974144    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"
Oct 31 03:51:39.974303 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.974187    4447 generic.go:356] "Generic (PLEG): container finished" podID="80bc3946-0385-4655-a0eb-3266a188f7ab" containerID="795351b27ca0ba3a922bcf786f45a2cfa79947e03f4a28921288e4a4a90b36d7" exitCode=0
Oct 31 03:51:39.974398 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.974371    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz" event={"ID":"80bc3946-0385-4655-a0eb-3266a188f7ab","Type":"ContainerDied","Data":"795351b27ca0ba3a922bcf786f45a2cfa79947e03f4a28921288e4a4a90b36d7"}
Oct 31 03:51:39.974479 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.974444    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz" event={"ID":"80bc3946-0385-4655-a0eb-3266a188f7ab","Type":"ContainerDied","Data":"50d6730da8def4ba8e959b60df66e4bba571899b8a4c40798f300adbc7b4ab17"}
Oct 31 03:51:39.981197 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.980859    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh" event={"ID":"dd66aaca-cbe4-414b-a056-551ed0c3f0b6","Type":"ContainerStarted","Data":"e0d9f58e6e6191486d5684e921dbac4c322090bd35d5470b1ee796ed4d033b59"}
Oct 31 03:51:39.981197 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.980955    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh" event={"ID":"dd66aaca-cbe4-414b-a056-551ed0c3f0b6","Type":"ContainerStarted","Data":"bb1ef9c9ecbb3dfc298a0aa31bf7345beddbd6fdb7ec0631e405edf2a91aaafe"}
Oct 31 03:51:39.985757 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.985731    4447 scope.go:122] "RemoveContainer" containerID="c07341a7a867c0bd3a6c0c4b635124ff8c096a1acc4f451e20ce43bf34b1feea"
Oct 31 03:51:39.986142 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:51:39.986109    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c07341a7a867c0bd3a6c0c4b635124ff8c096a1acc4f451e20ce43bf34b1feea\": container with ID starting with c07341a7a867c0bd3a6c0c4b635124ff8c096a1acc4f451e20ce43bf34b1feea not found: ID does not exist" containerID="c07341a7a867c0bd3a6c0c4b635124ff8c096a1acc4f451e20ce43bf34b1feea"
Oct 31 03:51:39.986248 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.986150    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c07341a7a867c0bd3a6c0c4b635124ff8c096a1acc4f451e20ce43bf34b1feea"} err="failed to get container status \"c07341a7a867c0bd3a6c0c4b635124ff8c096a1acc4f451e20ce43bf34b1feea\": rpc error: code = NotFound desc = could not find container \"c07341a7a867c0bd3a6c0c4b635124ff8c096a1acc4f451e20ce43bf34b1feea\": container with ID starting with c07341a7a867c0bd3a6c0c4b635124ff8c096a1acc4f451e20ce43bf34b1feea not found: ID does not exist"
Oct 31 03:51:39.986248 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:39.986177    4447 scope.go:122] "RemoveContainer" containerID="795351b27ca0ba3a922bcf786f45a2cfa79947e03f4a28921288e4a4a90b36d7"
Oct 31 03:51:40.029494 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:40.029476    4447 scope.go:122] "RemoveContainer" containerID="795351b27ca0ba3a922bcf786f45a2cfa79947e03f4a28921288e4a4a90b36d7"
Oct 31 03:51:40.029812 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:51:40.029793    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"795351b27ca0ba3a922bcf786f45a2cfa79947e03f4a28921288e4a4a90b36d7\": container with ID starting with 795351b27ca0ba3a922bcf786f45a2cfa79947e03f4a28921288e4a4a90b36d7 not found: ID does not exist" containerID="795351b27ca0ba3a922bcf786f45a2cfa79947e03f4a28921288e4a4a90b36d7"
Oct 31 03:51:40.029863 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:40.029822    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"795351b27ca0ba3a922bcf786f45a2cfa79947e03f4a28921288e4a4a90b36d7"} err="failed to get container status \"795351b27ca0ba3a922bcf786f45a2cfa79947e03f4a28921288e4a4a90b36d7\": rpc error: code = NotFound desc = could not find container \"795351b27ca0ba3a922bcf786f45a2cfa79947e03f4a28921288e4a4a90b36d7\": container with ID starting with 795351b27ca0ba3a922bcf786f45a2cfa79947e03f4a28921288e4a4a90b36d7 not found: ID does not exist"
Oct 31 03:51:40.139834 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:40.139800    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"]
Oct 31 03:51:40.191656 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:40.191618    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-5qvdz"]
Oct 31 03:51:40.255238 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:40.255161    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"]
Oct 31 03:51:40.260147 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:40.260115    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-096812-openshift-default-656f48f565-v8jr6"]
Oct 31 03:51:40.400966 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:40.400848    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh" podStartSLOduration=2.400823405 podStartE2EDuration="2.400823405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:51:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:51:40.391182169 +0000 UTC m=+1925.902164546" watchObservedRunningTime="2026-10-31 03:51:40.400823405 +0000 UTC m=+1925.911805784"
Oct 31 03:51:40.613509 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:40.613426    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5245fec2-9af5-4816-87de-e1d9a78930b2" path="/var/lib/kubelet/pods/5245fec2-9af5-4816-87de-e1d9a78930b2/volumes"
Oct 31 03:51:40.614061 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:40.614044    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80bc3946-0385-4655-a0eb-3266a188f7ab" path="/var/lib/kubelet/pods/80bc3946-0385-4655-a0eb-3266a188f7ab/volumes"
Oct 31 03:51:40.814769 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:40.814726    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:40.816586 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:40.816559    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 03:51:40.816715 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:40.816622    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh" podUID="dd66aaca-cbe4-414b-a056-551ed0c3f0b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 03:51:41.814755 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:41.814704    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 03:51:41.815257 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:41.814801    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh" podUID="dd66aaca-cbe4-414b-a056-551ed0c3f0b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 03:51:42.814671 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:42.814637    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 03:51:42.814855 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:42.814692    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh" podUID="dd66aaca-cbe4-414b-a056-551ed0c3f0b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 03:51:43.814651 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:43.814605    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 03:51:43.814845 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:43.814674    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh" podUID="dd66aaca-cbe4-414b-a056-551ed0c3f0b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 03:51:44.814659 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:44.814623    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 03:51:44.815109 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:44.814695    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh" podUID="dd66aaca-cbe4-414b-a056-551ed0c3f0b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 03:51:45.819103 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:45.819070    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:45.819588 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:45.819574    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:45.820282 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:45.820261    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:51:47.629179 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:47.629100    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-rcm2n"]
Oct 31 03:51:47.630242 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:47.630226    4447 cpu_manager.go:414] "containerMap: removing container" podUID="80bc3946-0385-4655-a0eb-3266a188f7ab" containerName="istio-proxy"
Oct 31 03:51:47.630285 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:47.630245    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80bc3946-0385-4655-a0eb-3266a188f7ab" containerName="istio-proxy"
Oct 31 03:51:47.630324 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:47.630293    4447 cpu_manager.go:414] "containerMap: removing container" podUID="5245fec2-9af5-4816-87de-e1d9a78930b2" containerName="istio-proxy"
Oct 31 03:51:47.630324 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:47.630301    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5245fec2-9af5-4816-87de-e1d9a78930b2" containerName="istio-proxy"
Oct 31 03:51:47.630464 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:47.630453    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="80bc3946-0385-4655-a0eb-3266a188f7ab" containerName="istio-proxy"
Oct 31 03:51:47.630494 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:47.630474    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="5245fec2-9af5-4816-87de-e1d9a78930b2" containerName="istio-proxy"
Oct 31 03:51:47.636092 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:47.636073    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-rcm2n"
Oct 31 03:51:47.690923 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:47.690876    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-rcm2n"]
Oct 31 03:51:47.705683 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:47.705648    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mktwq\" (UniqueName: \"kubernetes.io/projected/5c2f4dff-e3f0-4716-a107-aa20fb362160-kube-api-access-mktwq\") pod \"maas-api-revoke-keys-e2e-mt-1180d818-rcm2n\" (UID: \"5c2f4dff-e3f0-4716-a107-aa20fb362160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-rcm2n"
Oct 31 03:51:47.705959 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:47.705937    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c2f4dff-e3f0-4716-a107-aa20fb362160-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1180d818-rcm2n\" (UID: \"5c2f4dff-e3f0-4716-a107-aa20fb362160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-rcm2n"
Oct 31 03:51:47.706121 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:47.706104    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c2f4dff-e3f0-4716-a107-aa20fb362160-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1180d818-rcm2n\" (UID: \"5c2f4dff-e3f0-4716-a107-aa20fb362160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-rcm2n"
Oct 31 03:51:47.734104 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:47.734061    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:51:47.734380 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:47.734362    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 03:51:47.807448 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:47.807402    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mktwq\" (UniqueName: \"kubernetes.io/projected/5c2f4dff-e3f0-4716-a107-aa20fb362160-kube-api-access-mktwq\") pod \"maas-api-revoke-keys-e2e-mt-1180d818-rcm2n\" (UID: \"5c2f4dff-e3f0-4716-a107-aa20fb362160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-rcm2n"
Oct 31 03:51:47.807616 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:47.807459    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c2f4dff-e3f0-4716-a107-aa20fb362160-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1180d818-rcm2n\" (UID: \"5c2f4dff-e3f0-4716-a107-aa20fb362160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-rcm2n"
Oct 31 03:51:47.807616 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:47.807556    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c2f4dff-e3f0-4716-a107-aa20fb362160-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1180d818-rcm2n\" (UID: \"5c2f4dff-e3f0-4716-a107-aa20fb362160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-rcm2n"
Oct 31 03:51:47.808214 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:47.808187    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c2f4dff-e3f0-4716-a107-aa20fb362160-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1180d818-rcm2n\" (UID: \"5c2f4dff-e3f0-4716-a107-aa20fb362160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-rcm2n"
Oct 31 03:51:47.810017 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:47.809997    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c2f4dff-e3f0-4716-a107-aa20fb362160-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1180d818-rcm2n\" (UID: \"5c2f4dff-e3f0-4716-a107-aa20fb362160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-rcm2n"
Oct 31 03:51:47.868723 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:47.868694    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mktwq\" (UniqueName: \"kubernetes.io/projected/5c2f4dff-e3f0-4716-a107-aa20fb362160-kube-api-access-mktwq\") pod \"maas-api-revoke-keys-e2e-mt-1180d818-rcm2n\" (UID: \"5c2f4dff-e3f0-4716-a107-aa20fb362160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-rcm2n"
Oct 31 03:51:47.950453 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:47.950370    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-rcm2n"
Oct 31 03:51:48.335689 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:48.335645    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-rcm2n"]
Oct 31 03:51:48.335963 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:51:48.335942    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2f4dff_e3f0_4716_a107_aa20fb362160.slice/crio-14fdab666dd57229fc8d3ca45cf35b3d7304bae6a9af5a3259def7ad746b3d46 WatchSource:0}: Error finding container 14fdab666dd57229fc8d3ca45cf35b3d7304bae6a9af5a3259def7ad746b3d46: Status 404 returned error can't find the container with id 14fdab666dd57229fc8d3ca45cf35b3d7304bae6a9af5a3259def7ad746b3d46
Oct 31 03:51:48.961334 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:48.961309    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx"
Oct 31 03:51:48.961751 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:48.961428    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn"
Oct 31 03:51:49.003772 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:51:49.003740    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:51:49.076154 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:49.076120    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-rcm2n" event={"ID":"5c2f4dff-e3f0-4716-a107-aa20fb362160","Type":"ContainerStarted","Data":"723ca56d5cc404233a2b1865b6b758d6ac2a39c65696acaa27bfae766aabe009"}
Oct 31 03:51:49.076324 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:49.076171    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-rcm2n" event={"ID":"5c2f4dff-e3f0-4716-a107-aa20fb362160","Type":"ContainerStarted","Data":"14fdab666dd57229fc8d3ca45cf35b3d7304bae6a9af5a3259def7ad746b3d46"}
Oct 31 03:51:49.313159 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:49.313095    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-rcm2n" podStartSLOduration=2.313076329 podStartE2EDuration="2.313076329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:51:49.311386818 +0000 UTC m=+1934.822369193" watchObservedRunningTime="2026-10-31 03:51:49.313076329 +0000 UTC m=+1934.824058706"
Oct 31 03:51:50.085675 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:50.085640    4447 generic.go:356] "Generic (PLEG): container finished" podID="5c2f4dff-e3f0-4716-a107-aa20fb362160" containerID="723ca56d5cc404233a2b1865b6b758d6ac2a39c65696acaa27bfae766aabe009" exitCode=0
Oct 31 03:51:50.085675 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:50.085678    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-rcm2n" event={"ID":"5c2f4dff-e3f0-4716-a107-aa20fb362160","Type":"ContainerDied","Data":"723ca56d5cc404233a2b1865b6b758d6ac2a39c65696acaa27bfae766aabe009"}
Oct 31 03:51:51.256723 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:51.256699    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-rcm2n"
Oct 31 03:51:51.358796 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:51.358710    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c2f4dff-e3f0-4716-a107-aa20fb362160-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c2f4dff-e3f0-4716-a107-aa20fb362160-maas-api-openshift-ca\") pod \"5c2f4dff-e3f0-4716-a107-aa20fb362160\" (UID: \"5c2f4dff-e3f0-4716-a107-aa20fb362160\") "
Oct 31 03:51:51.358796 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:51.358791    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c2f4dff-e3f0-4716-a107-aa20fb362160-kube-api-access-mktwq\" (UniqueName: \"kubernetes.io/projected/5c2f4dff-e3f0-4716-a107-aa20fb362160-kube-api-access-mktwq\") pod \"5c2f4dff-e3f0-4716-a107-aa20fb362160\" (UID: \"5c2f4dff-e3f0-4716-a107-aa20fb362160\") "
Oct 31 03:51:51.359060 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:51.358842    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c2f4dff-e3f0-4716-a107-aa20fb362160-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c2f4dff-e3f0-4716-a107-aa20fb362160-maas-api-xks-ca\") pod \"5c2f4dff-e3f0-4716-a107-aa20fb362160\" (UID: \"5c2f4dff-e3f0-4716-a107-aa20fb362160\") "
Oct 31 03:51:51.359124 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:51.359101    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2f4dff-e3f0-4716-a107-aa20fb362160-maas-api-openshift-ca" pod "5c2f4dff-e3f0-4716-a107-aa20fb362160" (UID: "5c2f4dff-e3f0-4716-a107-aa20fb362160"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:51:51.359418 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:51.359391    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c2f4dff-e3f0-4716-a107-aa20fb362160-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:51.361155 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:51.361131    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c2f4dff-e3f0-4716-a107-aa20fb362160-maas-api-xks-ca" pod "5c2f4dff-e3f0-4716-a107-aa20fb362160" (UID: "5c2f4dff-e3f0-4716-a107-aa20fb362160"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:51:51.361259 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:51.361179    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2f4dff-e3f0-4716-a107-aa20fb362160-kube-api-access-mktwq" pod "5c2f4dff-e3f0-4716-a107-aa20fb362160" (UID: "5c2f4dff-e3f0-4716-a107-aa20fb362160"). InnerVolumeSpecName "kube-api-access-mktwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:51:51.460720 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:51.460686    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mktwq\" (UniqueName: \"kubernetes.io/projected/5c2f4dff-e3f0-4716-a107-aa20fb362160-kube-api-access-mktwq\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:51.460720 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:51.460716    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c2f4dff-e3f0-4716-a107-aa20fb362160-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:51:52.103056 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:52.103030    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-rcm2n"
Oct 31 03:51:52.103247 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:52.103062    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-rcm2n" event={"ID":"5c2f4dff-e3f0-4716-a107-aa20fb362160","Type":"ContainerDied","Data":"14fdab666dd57229fc8d3ca45cf35b3d7304bae6a9af5a3259def7ad746b3d46"}
Oct 31 03:51:52.103247 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:52.103104    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14fdab666dd57229fc8d3ca45cf35b3d7304bae6a9af5a3259def7ad746b3d46"
Oct 31 03:51:58.624926 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:51:58.624874    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc"]
Oct 31 03:52:03.625379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:03.625317    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc" podUID="0963d8e4-63ff-42d7-a0d8-f7c1bcc73761" containerName="payload-processing" containerID="cri-o://f316dc6c64dda0098a78e9a04b25114eb6e928eb76d6de5d0e2d253c1c10b318" gracePeriod=25
Oct 31 03:52:06.760830 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:06.760788    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc" podUID="0963d8e4-63ff-42d7-a0d8-f7c1bcc73761" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:52:08.835418 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:08.835387    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh"]
Oct 31 03:52:08.836245 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:08.836226    4447 cpu_manager.go:414] "containerMap: removing container" podUID="5c2f4dff-e3f0-4716-a107-aa20fb362160" containerName="revoke-keys"
Oct 31 03:52:08.836245 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:08.836245    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2f4dff-e3f0-4716-a107-aa20fb362160" containerName="revoke-keys"
Oct 31 03:52:08.836389 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:08.836346    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2f4dff-e3f0-4716-a107-aa20fb362160" containerName="revoke-keys"
Oct 31 03:52:08.840126 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:08.840111    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh"
Oct 31 03:52:08.892942 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:08.892850    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-05a9ff-dockercfg-lvf5g\""
Oct 31 03:52:09.032065 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:09.028807    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-05a9ff\""
Oct 31 03:52:09.034572 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:09.034174    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm7kn\" (UniqueName: \"kubernetes.io/projected/55f914e0-b8b7-403b-8ad0-1e4ef8d348dd-kube-api-access-sm7kn\") pod \"payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh\" (UID: \"55f914e0-b8b7-403b-8ad0-1e4ef8d348dd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh"
Oct 31 03:52:09.034572 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:09.034331    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55f914e0-b8b7-403b-8ad0-1e4ef8d348dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh\" (UID: \"55f914e0-b8b7-403b-8ad0-1e4ef8d348dd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh"
Oct 31 03:52:09.046005 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:09.045972    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh"]
Oct 31 03:52:09.046315 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:09.046296    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx"]
Oct 31 03:52:09.052294 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:09.052270    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx"
Oct 31 03:52:09.085288 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:09.085234    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx"]
Oct 31 03:52:09.136329 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:09.136234    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb6def7-2ef9-4cf2-974f-dc8f012eb431-plugins-config-volume\") pod \"payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx\" (UID: \"bfb6def7-2ef9-4cf2-974f-dc8f012eb431\") " pod="openshift-ingress/payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx"
Oct 31 03:52:09.136329 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:09.136275    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrn62\" (UniqueName: \"kubernetes.io/projected/bfb6def7-2ef9-4cf2-974f-dc8f012eb431-kube-api-access-vrn62\") pod \"payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx\" (UID: \"bfb6def7-2ef9-4cf2-974f-dc8f012eb431\") " pod="openshift-ingress/payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx"
Oct 31 03:52:09.136579 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:09.136440    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm7kn\" (UniqueName: \"kubernetes.io/projected/55f914e0-b8b7-403b-8ad0-1e4ef8d348dd-kube-api-access-sm7kn\") pod \"payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh\" (UID: \"55f914e0-b8b7-403b-8ad0-1e4ef8d348dd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh"
Oct 31 03:52:09.136579 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:09.136550    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55f914e0-b8b7-403b-8ad0-1e4ef8d348dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh\" (UID: \"55f914e0-b8b7-403b-8ad0-1e4ef8d348dd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh"
Oct 31 03:52:09.137261 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:09.137236    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55f914e0-b8b7-403b-8ad0-1e4ef8d348dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh\" (UID: \"55f914e0-b8b7-403b-8ad0-1e4ef8d348dd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh"
Oct 31 03:52:09.197482 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:09.197450    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm7kn\" (UniqueName: \"kubernetes.io/projected/55f914e0-b8b7-403b-8ad0-1e4ef8d348dd-kube-api-access-sm7kn\") pod \"payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh\" (UID: \"55f914e0-b8b7-403b-8ad0-1e4ef8d348dd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh"
Oct 31 03:52:09.237650 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:09.237609    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb6def7-2ef9-4cf2-974f-dc8f012eb431-plugins-config-volume\") pod \"payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx\" (UID: \"bfb6def7-2ef9-4cf2-974f-dc8f012eb431\") " pod="openshift-ingress/payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx"
Oct 31 03:52:09.237650 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:09.237651    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrn62\" (UniqueName: \"kubernetes.io/projected/bfb6def7-2ef9-4cf2-974f-dc8f012eb431-kube-api-access-vrn62\") pod \"payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx\" (UID: \"bfb6def7-2ef9-4cf2-974f-dc8f012eb431\") " pod="openshift-ingress/payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx"
Oct 31 03:52:09.238388 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:09.238366    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb6def7-2ef9-4cf2-974f-dc8f012eb431-plugins-config-volume\") pod \"payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx\" (UID: \"bfb6def7-2ef9-4cf2-974f-dc8f012eb431\") " pod="openshift-ingress/payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx"
Oct 31 03:52:09.299263 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:09.299232    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrn62\" (UniqueName: \"kubernetes.io/projected/bfb6def7-2ef9-4cf2-974f-dc8f012eb431-kube-api-access-vrn62\") pod \"payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx\" (UID: \"bfb6def7-2ef9-4cf2-974f-dc8f012eb431\") " pod="openshift-ingress/payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx"
Oct 31 03:52:09.396727 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:09.396635    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx"
Oct 31 03:52:09.454245 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:09.454215    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh"
Oct 31 03:52:09.869285 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:09.869246    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx"]
Oct 31 03:52:09.907678 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:52:09.907652    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb6def7_2ef9_4cf2_974f_dc8f012eb431.slice/crio-c5ea9acb14883e4b9dedd5ad461f625cee33fed6cc208eff3067a2c93ee181c8 WatchSource:0}: Error finding container c5ea9acb14883e4b9dedd5ad461f625cee33fed6cc208eff3067a2c93ee181c8: Status 404 returned error can't find the container with id c5ea9acb14883e4b9dedd5ad461f625cee33fed6cc208eff3067a2c93ee181c8
Oct 31 03:52:09.919288 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:09.919261    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh"]
Oct 31 03:52:09.922491 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:52:09.922469    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f914e0_b8b7_403b_8ad0_1e4ef8d348dd.slice/crio-311e0798b5a08fcfb4e52a9899510c289dbc09787a46490a36669d1a769598c3 WatchSource:0}: Error finding container 311e0798b5a08fcfb4e52a9899510c289dbc09787a46490a36669d1a769598c3: Status 404 returned error can't find the container with id 311e0798b5a08fcfb4e52a9899510c289dbc09787a46490a36669d1a769598c3
Oct 31 03:52:10.312931 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:10.312861    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh" event={"ID":"55f914e0-b8b7-403b-8ad0-1e4ef8d348dd","Type":"ContainerStarted","Data":"382e7568fab302ffdad09bfe207ae500bf55d1fd400cccd3a707e3d37c2b18bb"}
Oct 31 03:52:10.313126 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:10.312940    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh" event={"ID":"55f914e0-b8b7-403b-8ad0-1e4ef8d348dd","Type":"ContainerStarted","Data":"311e0798b5a08fcfb4e52a9899510c289dbc09787a46490a36669d1a769598c3"}
Oct 31 03:52:10.313126 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:10.312968    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh"
Oct 31 03:52:10.314783 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:10.314753    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx" event={"ID":"bfb6def7-2ef9-4cf2-974f-dc8f012eb431","Type":"ContainerStarted","Data":"cda7102f07320ac52b36e40a7e7dd594bae6acb8d2b46dad1dc60da2a6777ed4"}
Oct 31 03:52:10.314922 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:10.314804    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx"
Oct 31 03:52:10.314922 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:10.314826    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx" event={"ID":"bfb6def7-2ef9-4cf2-974f-dc8f012eb431","Type":"ContainerStarted","Data":"c5ea9acb14883e4b9dedd5ad461f625cee33fed6cc208eff3067a2c93ee181c8"}
Oct 31 03:52:10.431923 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:10.431848    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh" podStartSLOduration=2.431836013 podStartE2EDuration="2.431836013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:10.428011454 +0000 UTC m=+1955.938993831" watchObservedRunningTime="2026-10-31 03:52:10.431836013 +0000 UTC m=+1955.942818421"
Oct 31 03:52:16.761796 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:16.761759    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc" podUID="0963d8e4-63ff-42d7-a0d8-f7c1bcc73761" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:52:19.002994 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:52:19.002960    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:52:21.324790 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:21.324753    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh"
Oct 31 03:52:21.325228 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:21.325208    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx"
Oct 31 03:52:21.446573 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:21.446516    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx" podStartSLOduration=13.4465034 podStartE2EDuration="13.4465034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:10.588876263 +0000 UTC m=+1956.099858640" watchObservedRunningTime="2026-10-31 03:52:21.4465034 +0000 UTC m=+1966.957485776"
Oct 31 03:52:23.833050 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:23.833014    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"]
Oct 31 03:52:23.839066 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:23.839035    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:23.915202 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:23.915178    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc"
Oct 31 03:52:23.942364 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:23.942295    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"]
Oct 31 03:52:23.973523 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:23.973483    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cdmv\" (UniqueName: \"kubernetes.io/projected/24d4fdea-5469-4804-801f-3068de198f4a-kube-api-access-2cdmv\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:23.973720 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:23.973622    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24d4fdea-5469-4804-801f-3068de198f4a-istio-podinfo\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:23.973720 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:23.973662    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24d4fdea-5469-4804-801f-3068de198f4a-istiod-ca-cert\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:23.973843 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:23.973803    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-istio-data\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:23.973913 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:23.973840    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-credential-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:23.974028 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:23.974003    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24d4fdea-5469-4804-801f-3068de198f4a-istio-token\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:23.974079 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:23.974065    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-workload-certs\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:23.974175 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:23.974161    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-istio-envoy\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:23.974262 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:23.974247    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-workload-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:24.075848 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.074876    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0963d8e4-63ff-42d7-a0d8-f7c1bcc73761-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0963d8e4-63ff-42d7-a0d8-f7c1bcc73761-plugins-config-volume\") pod \"0963d8e4-63ff-42d7-a0d8-f7c1bcc73761\" (UID: \"0963d8e4-63ff-42d7-a0d8-f7c1bcc73761\") "
Oct 31 03:52:24.076132 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.075799    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0963d8e4-63ff-42d7-a0d8-f7c1bcc73761-plugins-config-volume" pod "0963d8e4-63ff-42d7-a0d8-f7c1bcc73761" (UID: "0963d8e4-63ff-42d7-a0d8-f7c1bcc73761"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:52:24.076132 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.075988    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0963d8e4-63ff-42d7-a0d8-f7c1bcc73761-kube-api-access-5jdfn\" (UniqueName: \"kubernetes.io/projected/0963d8e4-63ff-42d7-a0d8-f7c1bcc73761-kube-api-access-5jdfn\") pod \"0963d8e4-63ff-42d7-a0d8-f7c1bcc73761\" (UID: \"0963d8e4-63ff-42d7-a0d8-f7c1bcc73761\") "
Oct 31 03:52:24.076470 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.076442    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cdmv\" (UniqueName: \"kubernetes.io/projected/24d4fdea-5469-4804-801f-3068de198f4a-kube-api-access-2cdmv\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:24.076606 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.076518    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24d4fdea-5469-4804-801f-3068de198f4a-istio-podinfo\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:24.076606 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.076544    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24d4fdea-5469-4804-801f-3068de198f4a-istiod-ca-cert\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:24.076707 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.076634    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-istio-data\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:24.076707 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.076667    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-credential-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:24.076804 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.076718    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24d4fdea-5469-4804-801f-3068de198f4a-istio-token\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:24.076804 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.076747    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-workload-certs\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:24.076955 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.076815    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-istio-envoy\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:24.077611 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.077588    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-credential-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:24.078500 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.078162    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-istio-data\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:24.078985 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.078422    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-workload-certs\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:24.079123 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.078648    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-workload-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:24.079314 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.079300    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0963d8e4-63ff-42d7-a0d8-f7c1bcc73761-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:52:24.079410 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.078901    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-workload-socket\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:24.080100 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.080072    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24d4fdea-5469-4804-801f-3068de198f4a-istiod-ca-cert\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:24.080234 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.080209    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0963d8e4-63ff-42d7-a0d8-f7c1bcc73761-kube-api-access-5jdfn" pod "0963d8e4-63ff-42d7-a0d8-f7c1bcc73761" (UID: "0963d8e4-63ff-42d7-a0d8-f7c1bcc73761"). InnerVolumeSpecName "kube-api-access-5jdfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:24.087143 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.087112    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-istio-envoy\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:24.087527 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.087502    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24d4fdea-5469-4804-801f-3068de198f4a-istio-podinfo\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:24.148950 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.148834    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24d4fdea-5469-4804-801f-3068de198f4a-istio-token\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:24.151331 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.151301    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cdmv\" (UniqueName: \"kubernetes.io/projected/24d4fdea-5469-4804-801f-3068de198f4a-kube-api-access-2cdmv\") pod \"e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") " pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:24.155534 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.155510    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:24.180686 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.180659    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jdfn\" (UniqueName: \"kubernetes.io/projected/0963d8e4-63ff-42d7-a0d8-f7c1bcc73761-kube-api-access-5jdfn\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:52:24.432659 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.432558    4447 generic.go:356] "Generic (PLEG): container finished" podID="0963d8e4-63ff-42d7-a0d8-f7c1bcc73761" containerID="f316dc6c64dda0098a78e9a04b25114eb6e928eb76d6de5d0e2d253c1c10b318" exitCode=0
Oct 31 03:52:24.432659 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.432621    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc"
Oct 31 03:52:24.432659 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.432642    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc" event={"ID":"0963d8e4-63ff-42d7-a0d8-f7c1bcc73761","Type":"ContainerDied","Data":"f316dc6c64dda0098a78e9a04b25114eb6e928eb76d6de5d0e2d253c1c10b318"}
Oct 31 03:52:24.433002 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.432715    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc" event={"ID":"0963d8e4-63ff-42d7-a0d8-f7c1bcc73761","Type":"ContainerDied","Data":"3c102a11727d003022a6efe8598dcb7ca6bfc9635d3d6aaaeecbec1e66c8c37f"}
Oct 31 03:52:24.433002 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.432745    4447 scope.go:122] "RemoveContainer" containerID="f316dc6c64dda0098a78e9a04b25114eb6e928eb76d6de5d0e2d253c1c10b318"
Oct 31 03:52:24.444846 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.444828    4447 scope.go:122] "RemoveContainer" containerID="f316dc6c64dda0098a78e9a04b25114eb6e928eb76d6de5d0e2d253c1c10b318"
Oct 31 03:52:24.445139 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:52:24.445121    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f316dc6c64dda0098a78e9a04b25114eb6e928eb76d6de5d0e2d253c1c10b318\": container with ID starting with f316dc6c64dda0098a78e9a04b25114eb6e928eb76d6de5d0e2d253c1c10b318 not found: ID does not exist" containerID="f316dc6c64dda0098a78e9a04b25114eb6e928eb76d6de5d0e2d253c1c10b318"
Oct 31 03:52:24.445188 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.445148    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f316dc6c64dda0098a78e9a04b25114eb6e928eb76d6de5d0e2d253c1c10b318"} err="failed to get container status \"f316dc6c64dda0098a78e9a04b25114eb6e928eb76d6de5d0e2d253c1c10b318\": rpc error: code = NotFound desc = could not find container \"f316dc6c64dda0098a78e9a04b25114eb6e928eb76d6de5d0e2d253c1c10b318\": container with ID starting with f316dc6c64dda0098a78e9a04b25114eb6e928eb76d6de5d0e2d253c1c10b318 not found: ID does not exist"
Oct 31 03:52:24.591117 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.591088    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"]
Oct 31 03:52:24.591402 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:52:24.591382    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d4fdea_5469_4804_801f_3068de198f4a.slice/crio-37abd9b2d9372e965d27573baf37dbd14c215d67508b1812cbece54f568e9558 WatchSource:0}: Error finding container 37abd9b2d9372e965d27573baf37dbd14c215d67508b1812cbece54f568e9558: Status 404 returned error can't find the container with id 37abd9b2d9372e965d27573baf37dbd14c215d67508b1812cbece54f568e9558
Oct 31 03:52:24.594021 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.593989    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:52:24.594091 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.594061    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:52:24.594133 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.594106    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:52:24.600985 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.600959    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc"]
Oct 31 03:52:24.613101 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:24.613079    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-wvqtc"]
Oct 31 03:52:25.443128 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:25.443095    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd" event={"ID":"24d4fdea-5469-4804-801f-3068de198f4a","Type":"ContainerStarted","Data":"40bf1117e5e680a6b4766224d04ae4f90c592b41ff3bfa789009826cab4be49a"}
Oct 31 03:52:25.443617 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:25.443149    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd" event={"ID":"24d4fdea-5469-4804-801f-3068de198f4a","Type":"ContainerStarted","Data":"37abd9b2d9372e965d27573baf37dbd14c215d67508b1812cbece54f568e9558"}
Oct 31 03:52:25.567445 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:25.567378    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd" podStartSLOduration=2.567359358 podStartE2EDuration="2.567359358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:25.566039953 +0000 UTC m=+1971.077022342" watchObservedRunningTime="2026-10-31 03:52:25.567359358 +0000 UTC m=+1971.078341739"
Oct 31 03:52:26.156423 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:26.156395    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:26.157636 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:26.157620    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 31 03:52:26.157791 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:26.157773    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd" podUID="24d4fdea-5469-4804-801f-3068de198f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 31 03:52:26.610425 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:26.610391    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0963d8e4-63ff-42d7-a0d8-f7c1bcc73761" path="/var/lib/kubelet/pods/0963d8e4-63ff-42d7-a0d8-f7c1bcc73761/volumes"
Oct 31 03:52:27.156617 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:27.156579    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 31 03:52:27.156820 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:27.156653    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd" podUID="24d4fdea-5469-4804-801f-3068de198f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 31 03:52:28.017643 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:28.017609    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp"]
Oct 31 03:52:28.066879 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:28.066841    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-2accc9-djlcp"]
Oct 31 03:52:28.156857 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:28.156825    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 31 03:52:28.157042 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:28.156923    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd" podUID="24d4fdea-5469-4804-801f-3068de198f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 31 03:52:28.610588 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:28.610555    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d64cd46-dde8-4aae-ad8d-c11118d848e6" path="/var/lib/kubelet/pods/1d64cd46-dde8-4aae-ad8d-c11118d848e6/volumes"
Oct 31 03:52:29.156947 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:29.156871    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 31 03:52:29.157439 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:29.156965    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd" podUID="24d4fdea-5469-4804-801f-3068de198f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 31 03:52:30.156570 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:30.156542    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 31 03:52:30.156748 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:30.156589    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd" podUID="24d4fdea-5469-4804-801f-3068de198f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 31 03:52:31.161570 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:31.161537    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:31.162065 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:31.161904    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:31.162550 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:31.162530    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:52:36.381842 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:36.381803    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc"]
Oct 31 03:52:36.382655 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:36.382639    4447 cpu_manager.go:414] "containerMap: removing container" podUID="0963d8e4-63ff-42d7-a0d8-f7c1bcc73761" containerName="payload-processing"
Oct 31 03:52:36.382713 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:36.382656    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0963d8e4-63ff-42d7-a0d8-f7c1bcc73761" containerName="payload-processing"
Oct 31 03:52:36.382801 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:36.382792    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="0963d8e4-63ff-42d7-a0d8-f7c1bcc73761" containerName="payload-processing"
Oct 31 03:52:36.386992 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:36.386972    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc"
Oct 31 03:52:36.443816 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:36.443784    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc"]
Oct 31 03:52:36.488448 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:36.488419    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:52:36.488686 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:36.488505    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 03:52:36.515727 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:36.515695    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9f85dd-3164-45e4-b907-e8c9bc3a2b71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc\" (UID: \"6e9f85dd-3164-45e4-b907-e8c9bc3a2b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc"
Oct 31 03:52:36.515879 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:36.515846    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntckb\" (UniqueName: \"kubernetes.io/projected/6e9f85dd-3164-45e4-b907-e8c9bc3a2b71-kube-api-access-ntckb\") pod \"maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc\" (UID: \"6e9f85dd-3164-45e4-b907-e8c9bc3a2b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc"
Oct 31 03:52:36.516005 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:36.515987    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9f85dd-3164-45e4-b907-e8c9bc3a2b71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc\" (UID: \"6e9f85dd-3164-45e4-b907-e8c9bc3a2b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc"
Oct 31 03:52:36.624024 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:36.623991    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntckb\" (UniqueName: \"kubernetes.io/projected/6e9f85dd-3164-45e4-b907-e8c9bc3a2b71-kube-api-access-ntckb\") pod \"maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc\" (UID: \"6e9f85dd-3164-45e4-b907-e8c9bc3a2b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc"
Oct 31 03:52:36.624201 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:36.624063    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9f85dd-3164-45e4-b907-e8c9bc3a2b71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc\" (UID: \"6e9f85dd-3164-45e4-b907-e8c9bc3a2b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc"
Oct 31 03:52:36.624291 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:36.624250    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9f85dd-3164-45e4-b907-e8c9bc3a2b71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc\" (UID: \"6e9f85dd-3164-45e4-b907-e8c9bc3a2b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc"
Oct 31 03:52:36.624861 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:36.624836    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9f85dd-3164-45e4-b907-e8c9bc3a2b71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc\" (UID: \"6e9f85dd-3164-45e4-b907-e8c9bc3a2b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc"
Oct 31 03:52:36.635150 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:36.634770    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9f85dd-3164-45e4-b907-e8c9bc3a2b71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc\" (UID: \"6e9f85dd-3164-45e4-b907-e8c9bc3a2b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc"
Oct 31 03:52:36.685372 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:36.685335    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntckb\" (UniqueName: \"kubernetes.io/projected/6e9f85dd-3164-45e4-b907-e8c9bc3a2b71-kube-api-access-ntckb\") pod \"maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc\" (UID: \"6e9f85dd-3164-45e4-b907-e8c9bc3a2b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc"
Oct 31 03:52:36.708449 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:36.708413    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc"
Oct 31 03:52:37.099187 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:37.099153    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc"]
Oct 31 03:52:37.099685 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:52:37.099661    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9f85dd_3164_45e4_b907_e8c9bc3a2b71.slice/crio-f1cb67e86e5d96a3d68429f959edac102991b1621093e392789ff69b08096bda WatchSource:0}: Error finding container f1cb67e86e5d96a3d68429f959edac102991b1621093e392789ff69b08096bda: Status 404 returned error can't find the container with id f1cb67e86e5d96a3d68429f959edac102991b1621093e392789ff69b08096bda
Oct 31 03:52:37.587039 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:37.587005    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc" event={"ID":"6e9f85dd-3164-45e4-b907-e8c9bc3a2b71","Type":"ContainerStarted","Data":"460dda9937a8e041e6bbefeae2c8bbda525e0d4635938555bf3ac5486dad9cc4"}
Oct 31 03:52:37.587431 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:37.587059    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc" event={"ID":"6e9f85dd-3164-45e4-b907-e8c9bc3a2b71","Type":"ContainerStarted","Data":"f1cb67e86e5d96a3d68429f959edac102991b1621093e392789ff69b08096bda"}
Oct 31 03:52:37.706289 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:37.706224    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc" podStartSLOduration=1.706202413 podStartE2EDuration="1.706202413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:37.703526964 +0000 UTC m=+1983.214509341" watchObservedRunningTime="2026-10-31 03:52:37.706202413 +0000 UTC m=+1983.217184791"
Oct 31 03:52:38.599262 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:38.599231    4447 generic.go:356] "Generic (PLEG): container finished" podID="6e9f85dd-3164-45e4-b907-e8c9bc3a2b71" containerID="460dda9937a8e041e6bbefeae2c8bbda525e0d4635938555bf3ac5486dad9cc4" exitCode=0
Oct 31 03:52:38.599691 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:38.599313    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc" event={"ID":"6e9f85dd-3164-45e4-b907-e8c9bc3a2b71","Type":"ContainerDied","Data":"460dda9937a8e041e6bbefeae2c8bbda525e0d4635938555bf3ac5486dad9cc4"}
Oct 31 03:52:39.808122 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:39.808096    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc"
Oct 31 03:52:39.868583 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:39.868537    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e9f85dd-3164-45e4-b907-e8c9bc3a2b71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9f85dd-3164-45e4-b907-e8c9bc3a2b71-maas-api-openshift-ca\") pod \"6e9f85dd-3164-45e4-b907-e8c9bc3a2b71\" (UID: \"6e9f85dd-3164-45e4-b907-e8c9bc3a2b71\") "
Oct 31 03:52:39.868766 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:39.868602    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e9f85dd-3164-45e4-b907-e8c9bc3a2b71-kube-api-access-ntckb\" (UniqueName: \"kubernetes.io/projected/6e9f85dd-3164-45e4-b907-e8c9bc3a2b71-kube-api-access-ntckb\") pod \"6e9f85dd-3164-45e4-b907-e8c9bc3a2b71\" (UID: \"6e9f85dd-3164-45e4-b907-e8c9bc3a2b71\") "
Oct 31 03:52:39.868766 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:39.868685    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e9f85dd-3164-45e4-b907-e8c9bc3a2b71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9f85dd-3164-45e4-b907-e8c9bc3a2b71-maas-api-xks-ca\") pod \"6e9f85dd-3164-45e4-b907-e8c9bc3a2b71\" (UID: \"6e9f85dd-3164-45e4-b907-e8c9bc3a2b71\") "
Oct 31 03:52:39.869030 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:39.869006    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e9f85dd-3164-45e4-b907-e8c9bc3a2b71-maas-api-openshift-ca" pod "6e9f85dd-3164-45e4-b907-e8c9bc3a2b71" (UID: "6e9f85dd-3164-45e4-b907-e8c9bc3a2b71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:52:39.869515 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:39.869496    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9f85dd-3164-45e4-b907-e8c9bc3a2b71-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:52:39.871046 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:39.871014    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e9f85dd-3164-45e4-b907-e8c9bc3a2b71-maas-api-xks-ca" pod "6e9f85dd-3164-45e4-b907-e8c9bc3a2b71" (UID: "6e9f85dd-3164-45e4-b907-e8c9bc3a2b71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:52:39.871137 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:39.871083    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e9f85dd-3164-45e4-b907-e8c9bc3a2b71-kube-api-access-ntckb" pod "6e9f85dd-3164-45e4-b907-e8c9bc3a2b71" (UID: "6e9f85dd-3164-45e4-b907-e8c9bc3a2b71"). InnerVolumeSpecName "kube-api-access-ntckb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:39.971321 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:39.971274    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntckb\" (UniqueName: \"kubernetes.io/projected/6e9f85dd-3164-45e4-b907-e8c9bc3a2b71-kube-api-access-ntckb\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:52:39.971321 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:39.971317    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9f85dd-3164-45e4-b907-e8c9bc3a2b71-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:52:40.655213 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:40.655183    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc"
Oct 31 03:52:40.655386 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:40.655228    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc" event={"ID":"6e9f85dd-3164-45e4-b907-e8c9bc3a2b71","Type":"ContainerDied","Data":"f1cb67e86e5d96a3d68429f959edac102991b1621093e392789ff69b08096bda"}
Oct 31 03:52:40.655386 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:40.655275    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1cb67e86e5d96a3d68429f959edac102991b1621093e392789ff69b08096bda"
Oct 31 03:52:40.810046 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:40.810005    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"]
Oct 31 03:52:44.241641 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:44.241606    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ad7fb88-tcskb"]
Oct 31 03:52:44.242394 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:44.242379    4447 cpu_manager.go:414] "containerMap: removing container" podUID="6e9f85dd-3164-45e4-b907-e8c9bc3a2b71" containerName="revoke-keys"
Oct 31 03:52:44.242441 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:44.242395    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e9f85dd-3164-45e4-b907-e8c9bc3a2b71" containerName="revoke-keys"
Oct 31 03:52:44.242540 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:44.242531    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e9f85dd-3164-45e4-b907-e8c9bc3a2b71" containerName="revoke-keys"
Oct 31 03:52:44.246503 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:44.246485    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad7fb88-tcskb"
Oct 31 03:52:44.299241 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:44.299209    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 03:52:44.299464 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:44.299265    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:52:44.333335 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:44.333293    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5ch8\" (UniqueName: \"kubernetes.io/projected/ffe34fa0-f324-4227-a427-bf7228bfbecf-kube-api-access-m5ch8\") pod \"maas-api-revoke-sub-6ad7fb88-tcskb\" (UID: \"ffe34fa0-f324-4227-a427-bf7228bfbecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad7fb88-tcskb"
Oct 31 03:52:44.333523 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:44.333433    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe34fa0-f324-4227-a427-bf7228bfbecf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ad7fb88-tcskb\" (UID: \"ffe34fa0-f324-4227-a427-bf7228bfbecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad7fb88-tcskb"
Oct 31 03:52:44.333666 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:44.333594    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe34fa0-f324-4227-a427-bf7228bfbecf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ad7fb88-tcskb\" (UID: \"ffe34fa0-f324-4227-a427-bf7228bfbecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad7fb88-tcskb"
Oct 31 03:52:44.339314 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:44.339289    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ad7fb88-tcskb"]
Oct 31 03:52:44.434858 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:44.434824    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5ch8\" (UniqueName: \"kubernetes.io/projected/ffe34fa0-f324-4227-a427-bf7228bfbecf-kube-api-access-m5ch8\") pod \"maas-api-revoke-sub-6ad7fb88-tcskb\" (UID: \"ffe34fa0-f324-4227-a427-bf7228bfbecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad7fb88-tcskb"
Oct 31 03:52:44.435060 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:44.434903    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe34fa0-f324-4227-a427-bf7228bfbecf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ad7fb88-tcskb\" (UID: \"ffe34fa0-f324-4227-a427-bf7228bfbecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad7fb88-tcskb"
Oct 31 03:52:44.435060 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:44.435036    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe34fa0-f324-4227-a427-bf7228bfbecf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ad7fb88-tcskb\" (UID: \"ffe34fa0-f324-4227-a427-bf7228bfbecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad7fb88-tcskb"
Oct 31 03:52:44.435694 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:44.435669    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe34fa0-f324-4227-a427-bf7228bfbecf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ad7fb88-tcskb\" (UID: \"ffe34fa0-f324-4227-a427-bf7228bfbecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad7fb88-tcskb"
Oct 31 03:52:44.437542 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:44.437520    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe34fa0-f324-4227-a427-bf7228bfbecf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ad7fb88-tcskb\" (UID: \"ffe34fa0-f324-4227-a427-bf7228bfbecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad7fb88-tcskb"
Oct 31 03:52:44.496639 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:44.496573    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5ch8\" (UniqueName: \"kubernetes.io/projected/ffe34fa0-f324-4227-a427-bf7228bfbecf-kube-api-access-m5ch8\") pod \"maas-api-revoke-sub-6ad7fb88-tcskb\" (UID: \"ffe34fa0-f324-4227-a427-bf7228bfbecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad7fb88-tcskb"
Oct 31 03:52:44.560283 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:44.560246    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad7fb88-tcskb"
Oct 31 03:52:44.945059 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:44.945011    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ad7fb88-tcskb"]
Oct 31 03:52:44.945307 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:52:44.945111    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe34fa0_f324_4227_a427_bf7228bfbecf.slice/crio-7a87e33b7208c8de8943a190eede991a8fb46c0d73cd450d2f12d8d33e203b1a WatchSource:0}: Error finding container 7a87e33b7208c8de8943a190eede991a8fb46c0d73cd450d2f12d8d33e203b1a: Status 404 returned error can't find the container with id 7a87e33b7208c8de8943a190eede991a8fb46c0d73cd450d2f12d8d33e203b1a
Oct 31 03:52:45.699273 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:45.699191    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad7fb88-tcskb" event={"ID":"ffe34fa0-f324-4227-a427-bf7228bfbecf","Type":"ContainerStarted","Data":"ed78b34a0860a7f5b0e1019015bfe3c75157ae062976a9fa45d6a8d34d5180e5"}
Oct 31 03:52:45.699273 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:45.699243    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad7fb88-tcskb" event={"ID":"ffe34fa0-f324-4227-a427-bf7228bfbecf","Type":"ContainerStarted","Data":"7a87e33b7208c8de8943a190eede991a8fb46c0d73cd450d2f12d8d33e203b1a"}
Oct 31 03:52:45.816921 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:45.816846    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad7fb88-tcskb" podStartSLOduration=1.8168308359999998 podStartE2EDuration="1.816830836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:52:45.814939634 +0000 UTC m=+1991.325922012" watchObservedRunningTime="2026-10-31 03:52:45.816830836 +0000 UTC m=+1991.327813209"
Oct 31 03:52:46.709060 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:46.709029    4447 generic.go:356] "Generic (PLEG): container finished" podID="ffe34fa0-f324-4227-a427-bf7228bfbecf" containerID="ed78b34a0860a7f5b0e1019015bfe3c75157ae062976a9fa45d6a8d34d5180e5" exitCode=0
Oct 31 03:52:46.709519 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:46.709115    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad7fb88-tcskb" event={"ID":"ffe34fa0-f324-4227-a427-bf7228bfbecf","Type":"ContainerDied","Data":"ed78b34a0860a7f5b0e1019015bfe3c75157ae062976a9fa45d6a8d34d5180e5"}
Oct 31 03:52:47.864432 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:47.864399    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s"]
Oct 31 03:52:47.885088 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:47.885060    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad7fb88-tcskb"
Oct 31 03:52:47.977738 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:47.977644    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd"]
Oct 31 03:52:47.997447 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:47.997410    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffe34fa0-f324-4227-a427-bf7228bfbecf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe34fa0-f324-4227-a427-bf7228bfbecf-maas-api-xks-ca\") pod \"ffe34fa0-f324-4227-a427-bf7228bfbecf\" (UID: \"ffe34fa0-f324-4227-a427-bf7228bfbecf\") "
Oct 31 03:52:47.997641 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:47.997472    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffe34fa0-f324-4227-a427-bf7228bfbecf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe34fa0-f324-4227-a427-bf7228bfbecf-maas-api-openshift-ca\") pod \"ffe34fa0-f324-4227-a427-bf7228bfbecf\" (UID: \"ffe34fa0-f324-4227-a427-bf7228bfbecf\") "
Oct 31 03:52:47.997641 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:47.997590    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe34fa0-f324-4227-a427-bf7228bfbecf-kube-api-access-m5ch8\" (UniqueName: \"kubernetes.io/projected/ffe34fa0-f324-4227-a427-bf7228bfbecf-kube-api-access-m5ch8\") pod \"ffe34fa0-f324-4227-a427-bf7228bfbecf\" (UID: \"ffe34fa0-f324-4227-a427-bf7228bfbecf\") "
Oct 31 03:52:47.997865 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:47.997836    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffe34fa0-f324-4227-a427-bf7228bfbecf-maas-api-openshift-ca" pod "ffe34fa0-f324-4227-a427-bf7228bfbecf" (UID: "ffe34fa0-f324-4227-a427-bf7228bfbecf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:52:47.998174 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:47.998155    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe34fa0-f324-4227-a427-bf7228bfbecf-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:52:47.999805 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:47.999784    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffe34fa0-f324-4227-a427-bf7228bfbecf-maas-api-xks-ca" pod "ffe34fa0-f324-4227-a427-bf7228bfbecf" (UID: "ffe34fa0-f324-4227-a427-bf7228bfbecf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:52:47.999907 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:47.999820    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe34fa0-f324-4227-a427-bf7228bfbecf-kube-api-access-m5ch8" pod "ffe34fa0-f324-4227-a427-bf7228bfbecf" (UID: "ffe34fa0-f324-4227-a427-bf7228bfbecf"). InnerVolumeSpecName "kube-api-access-m5ch8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:48.099460 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:48.099428    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe34fa0-f324-4227-a427-bf7228bfbecf-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:52:48.099460 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:48.099460    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5ch8\" (UniqueName: \"kubernetes.io/projected/ffe34fa0-f324-4227-a427-bf7228bfbecf-kube-api-access-m5ch8\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:52:48.726378 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:48.726345    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad7fb88-tcskb"
Oct 31 03:52:48.726555 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:48.726372    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ad7fb88-tcskb" event={"ID":"ffe34fa0-f324-4227-a427-bf7228bfbecf","Type":"ContainerDied","Data":"7a87e33b7208c8de8943a190eede991a8fb46c0d73cd450d2f12d8d33e203b1a"}
Oct 31 03:52:48.726555 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:48.726422    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a87e33b7208c8de8943a190eede991a8fb46c0d73cd450d2f12d8d33e203b1a"
Oct 31 03:52:49.012976 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:52:49.012934    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:52:52.865497 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:52.865443    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s" podUID="21a7823a-b004-4b58-891e-bc463000308f" containerName="payload-processing" containerID="cri-o://d953e282332e56b84f9f7e676e0585cf49d623a30d31f2318dbfb05df24f7b2f" gracePeriod=25
Oct 31 03:52:52.984379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:52.981275    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd" podUID="a48bb0ee-3ca9-48f2-94e6-9b9d201a531a" containerName="payload-pre-processing" containerID="cri-o://6a5cd286fad0a1ad11f3f99660b6de1d383df65540230e139cef173c2702b9c7" gracePeriod=25
Oct 31 03:52:55.833602 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:55.833566    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main" containerID="cri-o://8b49f5b4b107f4eaa3cc38e556fdd7a376ae95efc408c134f56bcee71c77a7f0" gracePeriod=45
Oct 31 03:52:56.152592 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.152566    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:52:56.306676 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.306638    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-home\" (UniqueName: \"kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-home\") pod \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\" (UID: \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\") "
Oct 31 03:52:56.306874 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.306724    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-tls-certs\") pod \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\" (UID: \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\") "
Oct 31 03:52:56.306874 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.306806    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-tmp-dir\") pod \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\" (UID: \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\") "
Oct 31 03:52:56.307049 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.307029    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-kube-api-access-d8d2t\" (UniqueName: \"kubernetes.io/projected/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-kube-api-access-d8d2t\") pod \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\" (UID: \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\") "
Oct 31 03:52:56.307390 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.307145    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-tmp-dir" pod "3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" (UID: "3b3a27e8-9b2b-4ffb-8e92-a9af7df529de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:56.307497 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.307399    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-home" pod "3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" (UID: "3b3a27e8-9b2b-4ffb-8e92-a9af7df529de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:56.310197 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.310171    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-model-cache\") pod \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\" (UID: \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\") "
Oct 31 03:52:56.310303 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.310225    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-dshm\") pod \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\" (UID: \"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de\") "
Oct 31 03:52:56.310474 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.310424    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-model-cache" pod "3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" (UID: "3b3a27e8-9b2b-4ffb-8e92-a9af7df529de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:56.311038 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.311016    4447 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-tmp-dir\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:52:56.311143 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.311043    4447 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-model-cache\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:52:56.311143 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.311059    4447 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-home\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:52:56.311692 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.311639    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-kube-api-access-d8d2t" pod "3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" (UID: "3b3a27e8-9b2b-4ffb-8e92-a9af7df529de"). InnerVolumeSpecName "kube-api-access-d8d2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:56.311782 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.311730    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-tls-certs" pod "3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" (UID: "3b3a27e8-9b2b-4ffb-8e92-a9af7df529de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:52:56.313458 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.313430    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-dshm" pod "3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" (UID: "3b3a27e8-9b2b-4ffb-8e92-a9af7df529de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:56.412366 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.412294    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8d2t\" (UniqueName: \"kubernetes.io/projected/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-kube-api-access-d8d2t\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:52:56.412366 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.412319    4447 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-dshm\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:52:56.412366 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.412332    4447 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de-tls-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:52:56.816659 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.816627    4447 generic.go:356] "Generic (PLEG): container finished" podID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerID="8b49f5b4b107f4eaa3cc38e556fdd7a376ae95efc408c134f56bcee71c77a7f0" exitCode=0
Oct 31 03:52:56.816868 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.816671    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"
Oct 31 03:52:56.816868 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.816712    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" event={"ID":"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de","Type":"ContainerDied","Data":"8b49f5b4b107f4eaa3cc38e556fdd7a376ae95efc408c134f56bcee71c77a7f0"}
Oct 31 03:52:56.816868 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.816753    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z" event={"ID":"3b3a27e8-9b2b-4ffb-8e92-a9af7df529de","Type":"ContainerDied","Data":"1fbc89af967802bf6149f1f72907a10855ef1b2ef8e982af8d046d2c2d95714b"}
Oct 31 03:52:56.816868 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.816771    4447 scope.go:122] "RemoveContainer" containerID="8b49f5b4b107f4eaa3cc38e556fdd7a376ae95efc408c134f56bcee71c77a7f0"
Oct 31 03:52:56.827786 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.827770    4447 scope.go:122] "RemoveContainer" containerID="8b49f5b4b107f4eaa3cc38e556fdd7a376ae95efc408c134f56bcee71c77a7f0"
Oct 31 03:52:56.828092 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:52:56.828070    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b49f5b4b107f4eaa3cc38e556fdd7a376ae95efc408c134f56bcee71c77a7f0\": container with ID starting with 8b49f5b4b107f4eaa3cc38e556fdd7a376ae95efc408c134f56bcee71c77a7f0 not found: ID does not exist" containerID="8b49f5b4b107f4eaa3cc38e556fdd7a376ae95efc408c134f56bcee71c77a7f0"
Oct 31 03:52:56.828150 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.828100    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b49f5b4b107f4eaa3cc38e556fdd7a376ae95efc408c134f56bcee71c77a7f0"} err="failed to get container status \"8b49f5b4b107f4eaa3cc38e556fdd7a376ae95efc408c134f56bcee71c77a7f0\": rpc error: code = NotFound desc = could not find container \"8b49f5b4b107f4eaa3cc38e556fdd7a376ae95efc408c134f56bcee71c77a7f0\": container with ID starting with 8b49f5b4b107f4eaa3cc38e556fdd7a376ae95efc408c134f56bcee71c77a7f0 not found: ID does not exist"
Oct 31 03:52:56.984069 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:56.984034    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"]
Oct 31 03:52:57.033972 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:57.033938    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e00ad3/distinct-2-w1-e00ad3-kserve-86549b7664-z4j4z"]
Oct 31 03:52:58.771130 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:52:58.771083    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" path="/var/lib/kubelet/pods/3b3a27e8-9b2b-4ffb-8e92-a9af7df529de/volumes"
Oct 31 03:53:00.425947 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:00.425915    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s" podUID="21a7823a-b004-4b58-891e-bc463000308f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:53:00.426346 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:00.425907    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd" podUID="a48bb0ee-3ca9-48f2-94e6-9b9d201a531a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:53:02.278244 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.278210    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"]
Oct 31 03:53:02.280168 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.279635    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb" podUID="ce75e7a8-fe92-4059-8446-bffd5132c4b2" containerName="istio-proxy" containerID="cri-o://1836491a2f15993b44a3abebfdb979f398ee4d2f1aa4bee9f1145d1b98ff5d8c" gracePeriod=30
Oct 31 03:53:02.465262 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.465223    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq"]
Oct 31 03:53:02.466040 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.466025    4447 cpu_manager.go:414] "containerMap: removing container" podUID="ffe34fa0-f324-4227-a427-bf7228bfbecf" containerName="revoke-keys"
Oct 31 03:53:02.466130 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.466041    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe34fa0-f324-4227-a427-bf7228bfbecf" containerName="revoke-keys"
Oct 31 03:53:02.466130 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.466078    4447 cpu_manager.go:414] "containerMap: removing container" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main"
Oct 31 03:53:02.466130 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.466083    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main"
Oct 31 03:53:02.466324 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.466302    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe34fa0-f324-4227-a427-bf7228bfbecf" containerName="revoke-keys"
Oct 31 03:53:02.466413 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.466333    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b3a27e8-9b2b-4ffb-8e92-a9af7df529de" containerName="main"
Oct 31 03:53:02.470486 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.470469    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq"
Oct 31 03:53:02.528558 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.528485    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq"]
Oct 31 03:53:02.530864 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.530843    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b16de9\""
Oct 31 03:53:02.531074 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.531054    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b16de9-dockercfg-mwzbm\""
Oct 31 03:53:02.592915 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.592058    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82prv\" (UniqueName: \"kubernetes.io/projected/f3b51253-b4d7-49c9-be26-36a4e5320c82-kube-api-access-82prv\") pod \"payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq\" (UID: \"f3b51253-b4d7-49c9-be26-36a4e5320c82\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq"
Oct 31 03:53:02.592915 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.592257    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b51253-b4d7-49c9-be26-36a4e5320c82-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq\" (UID: \"f3b51253-b4d7-49c9-be26-36a4e5320c82\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq"
Oct 31 03:53:02.701938 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.698989    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82prv\" (UniqueName: \"kubernetes.io/projected/f3b51253-b4d7-49c9-be26-36a4e5320c82-kube-api-access-82prv\") pod \"payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq\" (UID: \"f3b51253-b4d7-49c9-be26-36a4e5320c82\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq"
Oct 31 03:53:02.701938 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.699126    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b51253-b4d7-49c9-be26-36a4e5320c82-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq\" (UID: \"f3b51253-b4d7-49c9-be26-36a4e5320c82\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq"
Oct 31 03:53:02.701938 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.701035    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b51253-b4d7-49c9-be26-36a4e5320c82-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq\" (UID: \"f3b51253-b4d7-49c9-be26-36a4e5320c82\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq"
Oct 31 03:53:02.718812 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.718780    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn"]
Oct 31 03:53:02.745159 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.745103    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn"
Oct 31 03:53:02.783011 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.780205    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82prv\" (UniqueName: \"kubernetes.io/projected/f3b51253-b4d7-49c9-be26-36a4e5320c82-kube-api-access-82prv\") pod \"payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq\" (UID: \"f3b51253-b4d7-49c9-be26-36a4e5320c82\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq"
Oct 31 03:53:02.784956 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.784934    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq"
Oct 31 03:53:02.788182 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.788156    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn"]
Oct 31 03:53:02.804753 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.804711    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/683a25bd-a64b-4a4c-8dbc-df6a1dd431aa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn\" (UID: \"683a25bd-a64b-4a4c-8dbc-df6a1dd431aa\") " pod="openshift-ingress/payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn"
Oct 31 03:53:02.804875 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.804803    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwgqk\" (UniqueName: \"kubernetes.io/projected/683a25bd-a64b-4a4c-8dbc-df6a1dd431aa-kube-api-access-wwgqk\") pod \"payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn\" (UID: \"683a25bd-a64b-4a4c-8dbc-df6a1dd431aa\") " pod="openshift-ingress/payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn"
Oct 31 03:53:02.906171 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.906117    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/683a25bd-a64b-4a4c-8dbc-df6a1dd431aa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn\" (UID: \"683a25bd-a64b-4a4c-8dbc-df6a1dd431aa\") " pod="openshift-ingress/payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn"
Oct 31 03:53:02.906349 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.906191    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwgqk\" (UniqueName: \"kubernetes.io/projected/683a25bd-a64b-4a4c-8dbc-df6a1dd431aa-kube-api-access-wwgqk\") pod \"payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn\" (UID: \"683a25bd-a64b-4a4c-8dbc-df6a1dd431aa\") " pod="openshift-ingress/payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn"
Oct 31 03:53:02.907362 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.907247    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/683a25bd-a64b-4a4c-8dbc-df6a1dd431aa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn\" (UID: \"683a25bd-a64b-4a4c-8dbc-df6a1dd431aa\") " pod="openshift-ingress/payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn"
Oct 31 03:53:02.967645 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:02.967605    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwgqk\" (UniqueName: \"kubernetes.io/projected/683a25bd-a64b-4a4c-8dbc-df6a1dd431aa-kube-api-access-wwgqk\") pod \"payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn\" (UID: \"683a25bd-a64b-4a4c-8dbc-df6a1dd431aa\") " pod="openshift-ingress/payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn"
Oct 31 03:53:03.138456 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:03.138375    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn"
Oct 31 03:53:03.180523 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:03.180494    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq"]
Oct 31 03:53:03.180650 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:53:03.180498    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b51253_b4d7_49c9_be26_36a4e5320c82.slice/crio-0fc31e09ecb7ea37250288ffd048c6dbbfba93ab5ecae81860069bf3e3ba21a8 WatchSource:0}: Error finding container 0fc31e09ecb7ea37250288ffd048c6dbbfba93ab5ecae81860069bf3e3ba21a8: Status 404 returned error can't find the container with id 0fc31e09ecb7ea37250288ffd048c6dbbfba93ab5ecae81860069bf3e3ba21a8
Oct 31 03:53:03.526368 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:03.526340    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn"]
Oct 31 03:53:03.526736 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:53:03.526706    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod683a25bd_a64b_4a4c_8dbc_df6a1dd431aa.slice/crio-893a3229e4d1c7f77eba7f81f03a1d51b08a86d7ca1996d8a69fbbd45e00a272 WatchSource:0}: Error finding container 893a3229e4d1c7f77eba7f81f03a1d51b08a86d7ca1996d8a69fbbd45e00a272: Status 404 returned error can't find the container with id 893a3229e4d1c7f77eba7f81f03a1d51b08a86d7ca1996d8a69fbbd45e00a272
Oct 31 03:53:03.540656 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:03.540628    4447 patch_prober.go:29] interesting pod/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:53:03.540771 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:03.540676    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb" podUID="ce75e7a8-fe92-4059-8446-bffd5132c4b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:53:03.902790 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:03.902679    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq" event={"ID":"f3b51253-b4d7-49c9-be26-36a4e5320c82","Type":"ContainerStarted","Data":"d91a31cbe8344153057de52fbc4cdd2b4d7c38669fbe3e232991ee0556734325"}
Oct 31 03:53:03.902790 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:03.902756    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq" event={"ID":"f3b51253-b4d7-49c9-be26-36a4e5320c82","Type":"ContainerStarted","Data":"0fc31e09ecb7ea37250288ffd048c6dbbfba93ab5ecae81860069bf3e3ba21a8"}
Oct 31 03:53:03.903074 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:03.902917    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq"
Oct 31 03:53:03.904425 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:03.904403    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn" event={"ID":"683a25bd-a64b-4a4c-8dbc-df6a1dd431aa","Type":"ContainerStarted","Data":"49c4dd056266b8b036bb0755ca7dc68d88fadffcf392bc151e27ebea4e268439"}
Oct 31 03:53:03.904556 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:03.904438    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn" event={"ID":"683a25bd-a64b-4a4c-8dbc-df6a1dd431aa","Type":"ContainerStarted","Data":"893a3229e4d1c7f77eba7f81f03a1d51b08a86d7ca1996d8a69fbbd45e00a272"}
Oct 31 03:53:03.904556 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:03.904456    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn"
Oct 31 03:53:04.027791 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:04.027727    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq" podStartSLOduration=2.027709851 podStartE2EDuration="2.027709851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:04.017933515 +0000 UTC m=+2009.528915892" watchObservedRunningTime="2026-10-31 03:53:04.027709851 +0000 UTC m=+2009.538692228"
Oct 31 03:53:04.155957 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:04.149994    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn" podStartSLOduration=2.149970569 podStartE2EDuration="2.149970569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:04.142425232 +0000 UTC m=+2009.653407609" watchObservedRunningTime="2026-10-31 03:53:04.149970569 +0000 UTC m=+2009.660952946"
Oct 31 03:53:07.555644 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.555578    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:53:07.683726 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.683694    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-data\") pod \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") "
Oct 31 03:53:07.683966 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.683738    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce75e7a8-fe92-4059-8446-bffd5132c4b2-kube-api-access-tmfgx\" (UniqueName: \"kubernetes.io/projected/ce75e7a8-fe92-4059-8446-bffd5132c4b2-kube-api-access-tmfgx\") pod \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") "
Oct 31 03:53:07.683966 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.683949    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istiod-ca-cert\") pod \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") "
Oct 31 03:53:07.684067 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.684039    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-data" pod "ce75e7a8-fe92-4059-8446-bffd5132c4b2" (UID: "ce75e7a8-fe92-4059-8446-bffd5132c4b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:53:07.684104 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.684057    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-envoy\") pod \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") "
Oct 31 03:53:07.684166 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.684119    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-credential-socket\") pod \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") "
Oct 31 03:53:07.684228 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.684213    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-token\" (UniqueName: \"kubernetes.io/projected/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-token\") pod \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") "
Oct 31 03:53:07.684312 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.684297    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-podinfo\") pod \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") "
Oct 31 03:53:07.684363 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.684324    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-credential-socket" pod "ce75e7a8-fe92-4059-8446-bffd5132c4b2" (UID: "ce75e7a8-fe92-4059-8446-bffd5132c4b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:53:07.684363 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.684264    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istiod-ca-cert" pod "ce75e7a8-fe92-4059-8446-bffd5132c4b2" (UID: "ce75e7a8-fe92-4059-8446-bffd5132c4b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:53:07.684363 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.684336    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-workload-socket\") pod \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") "
Oct 31 03:53:07.684507 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.684404    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-workload-certs\") pod \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\" (UID: \"ce75e7a8-fe92-4059-8446-bffd5132c4b2\") "
Oct 31 03:53:07.684599 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.684512    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-workload-socket" pod "ce75e7a8-fe92-4059-8446-bffd5132c4b2" (UID: "ce75e7a8-fe92-4059-8446-bffd5132c4b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:53:07.684772 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.684627    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-workload-certs" pod "ce75e7a8-fe92-4059-8446-bffd5132c4b2" (UID: "ce75e7a8-fe92-4059-8446-bffd5132c4b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:53:07.685356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.685328    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:53:07.685482 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.685360    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:53:07.685482 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.685375    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:53:07.685482 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.685388    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:53:07.685482 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.685402    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:53:07.686275 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.686255    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-envoy" pod "ce75e7a8-fe92-4059-8446-bffd5132c4b2" (UID: "ce75e7a8-fe92-4059-8446-bffd5132c4b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:53:07.686371 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.686355    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce75e7a8-fe92-4059-8446-bffd5132c4b2-kube-api-access-tmfgx" pod "ce75e7a8-fe92-4059-8446-bffd5132c4b2" (UID: "ce75e7a8-fe92-4059-8446-bffd5132c4b2"). InnerVolumeSpecName "kube-api-access-tmfgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:53:07.686438 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.686409    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-podinfo" pod "ce75e7a8-fe92-4059-8446-bffd5132c4b2" (UID: "ce75e7a8-fe92-4059-8446-bffd5132c4b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:53:07.686512 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.686497    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-token" pod "ce75e7a8-fe92-4059-8446-bffd5132c4b2" (UID: "ce75e7a8-fe92-4059-8446-bffd5132c4b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:53:07.787198 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.787164    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:53:07.787350 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.787202    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmfgx\" (UniqueName: \"kubernetes.io/projected/ce75e7a8-fe92-4059-8446-bffd5132c4b2-kube-api-access-tmfgx\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:53:07.787350 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.787220    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:53:07.787350 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.787235    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce75e7a8-fe92-4059-8446-bffd5132c4b2-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:53:07.939293 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.939211    4447 generic.go:356] "Generic (PLEG): container finished" podID="ce75e7a8-fe92-4059-8446-bffd5132c4b2" containerID="1836491a2f15993b44a3abebfdb979f398ee4d2f1aa4bee9f1145d1b98ff5d8c" exitCode=0
Oct 31 03:53:07.939293 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.939254    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"
Oct 31 03:53:07.939475 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.939292    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb" event={"ID":"ce75e7a8-fe92-4059-8446-bffd5132c4b2","Type":"ContainerDied","Data":"1836491a2f15993b44a3abebfdb979f398ee4d2f1aa4bee9f1145d1b98ff5d8c"}
Oct 31 03:53:07.939475 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.939347    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb" event={"ID":"ce75e7a8-fe92-4059-8446-bffd5132c4b2","Type":"ContainerDied","Data":"1b1e3f511759224d7ff1f4729caaa01d683a056097e6e197ee35c0b2ff90db1f"}
Oct 31 03:53:07.939475 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.939411    4447 scope.go:122] "RemoveContainer" containerID="1836491a2f15993b44a3abebfdb979f398ee4d2f1aa4bee9f1145d1b98ff5d8c"
Oct 31 03:53:07.951227 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.951207    4447 scope.go:122] "RemoveContainer" containerID="1836491a2f15993b44a3abebfdb979f398ee4d2f1aa4bee9f1145d1b98ff5d8c"
Oct 31 03:53:07.951514 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:53:07.951497    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1836491a2f15993b44a3abebfdb979f398ee4d2f1aa4bee9f1145d1b98ff5d8c\": container with ID starting with 1836491a2f15993b44a3abebfdb979f398ee4d2f1aa4bee9f1145d1b98ff5d8c not found: ID does not exist" containerID="1836491a2f15993b44a3abebfdb979f398ee4d2f1aa4bee9f1145d1b98ff5d8c"
Oct 31 03:53:07.951559 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:07.951522    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1836491a2f15993b44a3abebfdb979f398ee4d2f1aa4bee9f1145d1b98ff5d8c"} err="failed to get container status \"1836491a2f15993b44a3abebfdb979f398ee4d2f1aa4bee9f1145d1b98ff5d8c\": rpc error: code = NotFound desc = could not find container \"1836491a2f15993b44a3abebfdb979f398ee4d2f1aa4bee9f1145d1b98ff5d8c\": container with ID starting with 1836491a2f15993b44a3abebfdb979f398ee4d2f1aa4bee9f1145d1b98ff5d8c not found: ID does not exist"
Oct 31 03:53:08.106310 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:08.106277    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"]
Oct 31 03:53:08.110554 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:08.110527    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-4fclb"]
Oct 31 03:53:08.609958 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:08.609919    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce75e7a8-fe92-4059-8446-bffd5132c4b2" path="/var/lib/kubelet/pods/ce75e7a8-fe92-4059-8446-bffd5132c4b2/volumes"
Oct 31 03:53:10.426491 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:10.426459    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s" podUID="21a7823a-b004-4b58-891e-bc463000308f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:53:10.426491 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:10.426487    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd" podUID="a48bb0ee-3ca9-48f2-94e6-9b9d201a531a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:53:13.134603 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.134543    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s"
Oct 31 03:53:13.262714 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.262687    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a7823a-b004-4b58-891e-bc463000308f-kube-api-access-w4dd4\" (UniqueName: \"kubernetes.io/projected/21a7823a-b004-4b58-891e-bc463000308f-kube-api-access-w4dd4\") pod \"21a7823a-b004-4b58-891e-bc463000308f\" (UID: \"21a7823a-b004-4b58-891e-bc463000308f\") "
Oct 31 03:53:13.262914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.262738    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21a7823a-b004-4b58-891e-bc463000308f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a7823a-b004-4b58-891e-bc463000308f-plugins-config-volume\") pod \"21a7823a-b004-4b58-891e-bc463000308f\" (UID: \"21a7823a-b004-4b58-891e-bc463000308f\") "
Oct 31 03:53:13.263154 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.263125    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a7823a-b004-4b58-891e-bc463000308f-plugins-config-volume" pod "21a7823a-b004-4b58-891e-bc463000308f" (UID: "21a7823a-b004-4b58-891e-bc463000308f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:53:13.263335 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.263318    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a7823a-b004-4b58-891e-bc463000308f-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:53:13.264560 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.264538    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd"
Oct 31 03:53:13.265063 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.265041    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a7823a-b004-4b58-891e-bc463000308f-kube-api-access-w4dd4" pod "21a7823a-b004-4b58-891e-bc463000308f" (UID: "21a7823a-b004-4b58-891e-bc463000308f"). InnerVolumeSpecName "kube-api-access-w4dd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:53:13.364635 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.364602    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a48bb0ee-3ca9-48f2-94e6-9b9d201a531a-kube-api-access-fxkbl\" (UniqueName: \"kubernetes.io/projected/a48bb0ee-3ca9-48f2-94e6-9b9d201a531a-kube-api-access-fxkbl\") pod \"a48bb0ee-3ca9-48f2-94e6-9b9d201a531a\" (UID: \"a48bb0ee-3ca9-48f2-94e6-9b9d201a531a\") "
Oct 31 03:53:13.364796 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.364658    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a48bb0ee-3ca9-48f2-94e6-9b9d201a531a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48bb0ee-3ca9-48f2-94e6-9b9d201a531a-plugins-config-volume\") pod \"a48bb0ee-3ca9-48f2-94e6-9b9d201a531a\" (UID: \"a48bb0ee-3ca9-48f2-94e6-9b9d201a531a\") "
Oct 31 03:53:13.365086 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.365061    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a48bb0ee-3ca9-48f2-94e6-9b9d201a531a-plugins-config-volume" pod "a48bb0ee-3ca9-48f2-94e6-9b9d201a531a" (UID: "a48bb0ee-3ca9-48f2-94e6-9b9d201a531a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:53:13.365234 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.365219    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4dd4\" (UniqueName: \"kubernetes.io/projected/21a7823a-b004-4b58-891e-bc463000308f-kube-api-access-w4dd4\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:53:13.365301 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.365235    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48bb0ee-3ca9-48f2-94e6-9b9d201a531a-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:53:13.366868 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.366843    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48bb0ee-3ca9-48f2-94e6-9b9d201a531a-kube-api-access-fxkbl" pod "a48bb0ee-3ca9-48f2-94e6-9b9d201a531a" (UID: "a48bb0ee-3ca9-48f2-94e6-9b9d201a531a"). InnerVolumeSpecName "kube-api-access-fxkbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:53:13.467282 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.467238    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxkbl\" (UniqueName: \"kubernetes.io/projected/a48bb0ee-3ca9-48f2-94e6-9b9d201a531a-kube-api-access-fxkbl\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:53:13.986942 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.986907    4447 generic.go:356] "Generic (PLEG): container finished" podID="a48bb0ee-3ca9-48f2-94e6-9b9d201a531a" containerID="6a5cd286fad0a1ad11f3f99660b6de1d383df65540230e139cef173c2702b9c7" exitCode=0
Oct 31 03:53:13.986942 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.986943    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd"
Oct 31 03:53:13.987188 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.986977    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd" event={"ID":"a48bb0ee-3ca9-48f2-94e6-9b9d201a531a","Type":"ContainerDied","Data":"6a5cd286fad0a1ad11f3f99660b6de1d383df65540230e139cef173c2702b9c7"}
Oct 31 03:53:13.987188 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.987025    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd" event={"ID":"a48bb0ee-3ca9-48f2-94e6-9b9d201a531a","Type":"ContainerDied","Data":"9c705dd6d5ded6d56fec93b9bb992deb66cbbfc9d1a44efca42c40f9bd0a74a5"}
Oct 31 03:53:13.987188 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.987049    4447 scope.go:122] "RemoveContainer" containerID="6a5cd286fad0a1ad11f3f99660b6de1d383df65540230e139cef173c2702b9c7"
Oct 31 03:53:13.988573 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.988547    4447 generic.go:356] "Generic (PLEG): container finished" podID="21a7823a-b004-4b58-891e-bc463000308f" containerID="d953e282332e56b84f9f7e676e0585cf49d623a30d31f2318dbfb05df24f7b2f" exitCode=0
Oct 31 03:53:13.988866 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.988592    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s"
Oct 31 03:53:13.988866 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.988601    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s" event={"ID":"21a7823a-b004-4b58-891e-bc463000308f","Type":"ContainerDied","Data":"d953e282332e56b84f9f7e676e0585cf49d623a30d31f2318dbfb05df24f7b2f"}
Oct 31 03:53:13.988866 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.988634    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s" event={"ID":"21a7823a-b004-4b58-891e-bc463000308f","Type":"ContainerDied","Data":"67918b335dbf391b420e868ef5758ced253c1aa195c877576f2288ce11fdf17a"}
Oct 31 03:53:13.999660 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.999643    4447 scope.go:122] "RemoveContainer" containerID="6a5cd286fad0a1ad11f3f99660b6de1d383df65540230e139cef173c2702b9c7"
Oct 31 03:53:13.999955 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:53:13.999938    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a5cd286fad0a1ad11f3f99660b6de1d383df65540230e139cef173c2702b9c7\": container with ID starting with 6a5cd286fad0a1ad11f3f99660b6de1d383df65540230e139cef173c2702b9c7 not found: ID does not exist" containerID="6a5cd286fad0a1ad11f3f99660b6de1d383df65540230e139cef173c2702b9c7"
Oct 31 03:53:14.000016 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.999964    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a5cd286fad0a1ad11f3f99660b6de1d383df65540230e139cef173c2702b9c7"} err="failed to get container status \"6a5cd286fad0a1ad11f3f99660b6de1d383df65540230e139cef173c2702b9c7\": rpc error: code = NotFound desc = could not find container \"6a5cd286fad0a1ad11f3f99660b6de1d383df65540230e139cef173c2702b9c7\": container with ID starting with 6a5cd286fad0a1ad11f3f99660b6de1d383df65540230e139cef173c2702b9c7 not found: ID does not exist"
Oct 31 03:53:14.000016 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:13.999981    4447 scope.go:122] "RemoveContainer" containerID="d953e282332e56b84f9f7e676e0585cf49d623a30d31f2318dbfb05df24f7b2f"
Oct 31 03:53:14.009701 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:14.009685    4447 scope.go:122] "RemoveContainer" containerID="d953e282332e56b84f9f7e676e0585cf49d623a30d31f2318dbfb05df24f7b2f"
Oct 31 03:53:14.010010 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:53:14.009980    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d953e282332e56b84f9f7e676e0585cf49d623a30d31f2318dbfb05df24f7b2f\": container with ID starting with d953e282332e56b84f9f7e676e0585cf49d623a30d31f2318dbfb05df24f7b2f not found: ID does not exist" containerID="d953e282332e56b84f9f7e676e0585cf49d623a30d31f2318dbfb05df24f7b2f"
Oct 31 03:53:14.010104 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:14.010010    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d953e282332e56b84f9f7e676e0585cf49d623a30d31f2318dbfb05df24f7b2f"} err="failed to get container status \"d953e282332e56b84f9f7e676e0585cf49d623a30d31f2318dbfb05df24f7b2f\": rpc error: code = NotFound desc = could not find container \"d953e282332e56b84f9f7e676e0585cf49d623a30d31f2318dbfb05df24f7b2f\": container with ID starting with d953e282332e56b84f9f7e676e0585cf49d623a30d31f2318dbfb05df24f7b2f not found: ID does not exist"
Oct 31 03:53:14.154113 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:14.154031    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s"]
Oct 31 03:53:14.204172 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:14.204132    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e00ad3-5f9c44fd7-jvl8s"]
Oct 31 03:53:14.267020 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:14.266991    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd"]
Oct 31 03:53:14.271033 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:14.271006    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e00ad3-6d785dddf5-qz7jd"]
Oct 31 03:53:14.613143 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:14.613103    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21a7823a-b004-4b58-891e-bc463000308f" path="/var/lib/kubelet/pods/21a7823a-b004-4b58-891e-bc463000308f/volumes"
Oct 31 03:53:14.613605 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:14.613583    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a48bb0ee-3ca9-48f2-94e6-9b9d201a531a" path="/var/lib/kubelet/pods/a48bb0ee-3ca9-48f2-94e6-9b9d201a531a/volumes"
Oct 31 03:53:14.915347 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:14.915271    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn"
Oct 31 03:53:14.915347 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:14.915334    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq"
Oct 31 03:53:16.421655 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.421567    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d114f70b-lcntl"]
Oct 31 03:53:16.422350 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.422335    4447 cpu_manager.go:414] "containerMap: removing container" podUID="ce75e7a8-fe92-4059-8446-bffd5132c4b2" containerName="istio-proxy"
Oct 31 03:53:16.422413 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.422352    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce75e7a8-fe92-4059-8446-bffd5132c4b2" containerName="istio-proxy"
Oct 31 03:53:16.422413 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.422374    4447 cpu_manager.go:414] "containerMap: removing container" podUID="a48bb0ee-3ca9-48f2-94e6-9b9d201a531a" containerName="payload-pre-processing"
Oct 31 03:53:16.422413 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.422379    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a48bb0ee-3ca9-48f2-94e6-9b9d201a531a" containerName="payload-pre-processing"
Oct 31 03:53:16.422413 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.422406    4447 cpu_manager.go:414] "containerMap: removing container" podUID="21a7823a-b004-4b58-891e-bc463000308f" containerName="payload-processing"
Oct 31 03:53:16.422413 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.422412    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21a7823a-b004-4b58-891e-bc463000308f" containerName="payload-processing"
Oct 31 03:53:16.422570 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.422519    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="21a7823a-b004-4b58-891e-bc463000308f" containerName="payload-processing"
Oct 31 03:53:16.422570 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.422531    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce75e7a8-fe92-4059-8446-bffd5132c4b2" containerName="istio-proxy"
Oct 31 03:53:16.422570 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.422541    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="a48bb0ee-3ca9-48f2-94e6-9b9d201a531a" containerName="payload-pre-processing"
Oct 31 03:53:16.427773 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.427756    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d114f70b-lcntl"
Oct 31 03:53:16.476216 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.476187    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba367b2-kkk7d"]
Oct 31 03:53:16.480919 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.480867    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba367b2-kkk7d"
Oct 31 03:53:16.483711 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.483687    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d114f70b-lcntl"]
Oct 31 03:53:16.519334 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.519300    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7cvc\" (UniqueName: \"kubernetes.io/projected/0cb4f9a2-dfba-43a6-8d42-a68281d62d37-kube-api-access-h7cvc\") pod \"maas-api-revoke-sub-d114f70b-lcntl\" (UID: \"0cb4f9a2-dfba-43a6-8d42-a68281d62d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d114f70b-lcntl"
Oct 31 03:53:16.519518 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.519355    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb4f9a2-dfba-43a6-8d42-a68281d62d37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d114f70b-lcntl\" (UID: \"0cb4f9a2-dfba-43a6-8d42-a68281d62d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d114f70b-lcntl"
Oct 31 03:53:16.519518 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.519476    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb4f9a2-dfba-43a6-8d42-a68281d62d37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d114f70b-lcntl\" (UID: \"0cb4f9a2-dfba-43a6-8d42-a68281d62d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d114f70b-lcntl"
Oct 31 03:53:16.521573 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.521545    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:53:16.521701 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.521597    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 03:53:16.541876 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.541841    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba367b2-kkk7d"]
Oct 31 03:53:16.620687 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.620649    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpxx8\" (UniqueName: \"kubernetes.io/projected/0ea92bdf-3a68-49be-80fe-b32ea7e220a2-kube-api-access-lpxx8\") pod \"maas-api-revoke-sub-2ba367b2-kkk7d\" (UID: \"0ea92bdf-3a68-49be-80fe-b32ea7e220a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba367b2-kkk7d"
Oct 31 03:53:16.620849 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.620710    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7cvc\" (UniqueName: \"kubernetes.io/projected/0cb4f9a2-dfba-43a6-8d42-a68281d62d37-kube-api-access-h7cvc\") pod \"maas-api-revoke-sub-d114f70b-lcntl\" (UID: \"0cb4f9a2-dfba-43a6-8d42-a68281d62d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d114f70b-lcntl"
Oct 31 03:53:16.620849 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.620775    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb4f9a2-dfba-43a6-8d42-a68281d62d37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d114f70b-lcntl\" (UID: \"0cb4f9a2-dfba-43a6-8d42-a68281d62d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d114f70b-lcntl"
Oct 31 03:53:16.620970 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.620912    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb4f9a2-dfba-43a6-8d42-a68281d62d37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d114f70b-lcntl\" (UID: \"0cb4f9a2-dfba-43a6-8d42-a68281d62d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d114f70b-lcntl"
Oct 31 03:53:16.621019 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.620987    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea92bdf-3a68-49be-80fe-b32ea7e220a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ba367b2-kkk7d\" (UID: \"0ea92bdf-3a68-49be-80fe-b32ea7e220a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba367b2-kkk7d"
Oct 31 03:53:16.621070 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.621044    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea92bdf-3a68-49be-80fe-b32ea7e220a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ba367b2-kkk7d\" (UID: \"0ea92bdf-3a68-49be-80fe-b32ea7e220a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba367b2-kkk7d"
Oct 31 03:53:16.621540 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.621516    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb4f9a2-dfba-43a6-8d42-a68281d62d37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d114f70b-lcntl\" (UID: \"0cb4f9a2-dfba-43a6-8d42-a68281d62d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d114f70b-lcntl"
Oct 31 03:53:16.623454 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.623433    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb4f9a2-dfba-43a6-8d42-a68281d62d37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d114f70b-lcntl\" (UID: \"0cb4f9a2-dfba-43a6-8d42-a68281d62d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d114f70b-lcntl"
Oct 31 03:53:16.683048 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.682977    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7cvc\" (UniqueName: \"kubernetes.io/projected/0cb4f9a2-dfba-43a6-8d42-a68281d62d37-kube-api-access-h7cvc\") pod \"maas-api-revoke-sub-d114f70b-lcntl\" (UID: \"0cb4f9a2-dfba-43a6-8d42-a68281d62d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d114f70b-lcntl"
Oct 31 03:53:16.722601 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.722572    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea92bdf-3a68-49be-80fe-b32ea7e220a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ba367b2-kkk7d\" (UID: \"0ea92bdf-3a68-49be-80fe-b32ea7e220a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba367b2-kkk7d"
Oct 31 03:53:16.722755 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.722633    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea92bdf-3a68-49be-80fe-b32ea7e220a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ba367b2-kkk7d\" (UID: \"0ea92bdf-3a68-49be-80fe-b32ea7e220a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba367b2-kkk7d"
Oct 31 03:53:16.722755 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.722694    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpxx8\" (UniqueName: \"kubernetes.io/projected/0ea92bdf-3a68-49be-80fe-b32ea7e220a2-kube-api-access-lpxx8\") pod \"maas-api-revoke-sub-2ba367b2-kkk7d\" (UID: \"0ea92bdf-3a68-49be-80fe-b32ea7e220a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba367b2-kkk7d"
Oct 31 03:53:16.723433 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.723358    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea92bdf-3a68-49be-80fe-b32ea7e220a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ba367b2-kkk7d\" (UID: \"0ea92bdf-3a68-49be-80fe-b32ea7e220a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba367b2-kkk7d"
Oct 31 03:53:16.725034 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.725012    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea92bdf-3a68-49be-80fe-b32ea7e220a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ba367b2-kkk7d\" (UID: \"0ea92bdf-3a68-49be-80fe-b32ea7e220a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba367b2-kkk7d"
Oct 31 03:53:16.741744 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.741722    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d114f70b-lcntl"
Oct 31 03:53:16.785238 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.785202    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpxx8\" (UniqueName: \"kubernetes.io/projected/0ea92bdf-3a68-49be-80fe-b32ea7e220a2-kube-api-access-lpxx8\") pod \"maas-api-revoke-sub-2ba367b2-kkk7d\" (UID: \"0ea92bdf-3a68-49be-80fe-b32ea7e220a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba367b2-kkk7d"
Oct 31 03:53:16.795791 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:16.795764    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba367b2-kkk7d"
Oct 31 03:53:17.124318 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:17.124287    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d114f70b-lcntl"]
Oct 31 03:53:17.124631 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:53:17.124610    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb4f9a2_dfba_43a6_8d42_a68281d62d37.slice/crio-ac27194d804d9c35dacf909f5f1a412bf30a4abffe295ab4f0964db5191bc714 WatchSource:0}: Error finding container ac27194d804d9c35dacf909f5f1a412bf30a4abffe295ab4f0964db5191bc714: Status 404 returned error can't find the container with id ac27194d804d9c35dacf909f5f1a412bf30a4abffe295ab4f0964db5191bc714
Oct 31 03:53:17.228710 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:17.228677    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba367b2-kkk7d"]
Oct 31 03:53:18.022657 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:18.022621    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba367b2-kkk7d" event={"ID":"0ea92bdf-3a68-49be-80fe-b32ea7e220a2","Type":"ContainerStarted","Data":"d03beac4c3f556b9e90b4535dcd1354442b991728fa67184389b5db869501e6e"}
Oct 31 03:53:18.023141 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:18.022685    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba367b2-kkk7d" event={"ID":"0ea92bdf-3a68-49be-80fe-b32ea7e220a2","Type":"ContainerStarted","Data":"12e237fe6d8e74a6de983e268d8cae6824fc030e7f76930923bc28e24ae8c389"}
Oct 31 03:53:18.024568 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:18.024544    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d114f70b-lcntl" event={"ID":"0cb4f9a2-dfba-43a6-8d42-a68281d62d37","Type":"ContainerStarted","Data":"1c441a63dacf0748f040570cf32a76fa8109693ce7430e475e695978551aca87"}
Oct 31 03:53:18.024699 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:18.024584    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d114f70b-lcntl" event={"ID":"0cb4f9a2-dfba-43a6-8d42-a68281d62d37","Type":"ContainerStarted","Data":"ac27194d804d9c35dacf909f5f1a412bf30a4abffe295ab4f0964db5191bc714"}
Oct 31 03:53:18.145635 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:18.145587    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba367b2-kkk7d" podStartSLOduration=2.145570891 podStartE2EDuration="2.145570891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:18.140722388 +0000 UTC m=+2023.651704766" watchObservedRunningTime="2026-10-31 03:53:18.145570891 +0000 UTC m=+2023.656553267"
Oct 31 03:53:18.260517 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:18.260447    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d114f70b-lcntl" podStartSLOduration=2.260427492 podStartE2EDuration="2.260427492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:18.25734596 +0000 UTC m=+2023.768328341" watchObservedRunningTime="2026-10-31 03:53:18.260427492 +0000 UTC m=+2023.771409869"
Oct 31 03:53:19.029910 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:53:19.020698    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:53:19.045558 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:19.045511    4447 generic.go:356] "Generic (PLEG): container finished" podID="0ea92bdf-3a68-49be-80fe-b32ea7e220a2" containerID="d03beac4c3f556b9e90b4535dcd1354442b991728fa67184389b5db869501e6e" exitCode=0
Oct 31 03:53:19.045751 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:19.045652    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba367b2-kkk7d" event={"ID":"0ea92bdf-3a68-49be-80fe-b32ea7e220a2","Type":"ContainerDied","Data":"d03beac4c3f556b9e90b4535dcd1354442b991728fa67184389b5db869501e6e"}
Oct 31 03:53:19.047685 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:19.047654    4447 generic.go:356] "Generic (PLEG): container finished" podID="0cb4f9a2-dfba-43a6-8d42-a68281d62d37" containerID="1c441a63dacf0748f040570cf32a76fa8109693ce7430e475e695978551aca87" exitCode=0
Oct 31 03:53:19.047842 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:19.047742    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d114f70b-lcntl" event={"ID":"0cb4f9a2-dfba-43a6-8d42-a68281d62d37","Type":"ContainerDied","Data":"1c441a63dacf0748f040570cf32a76fa8109693ce7430e475e695978551aca87"}
Oct 31 03:53:20.260214 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:20.260186    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d114f70b-lcntl"
Oct 31 03:53:20.264031 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:20.264010    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba367b2-kkk7d"
Oct 31 03:53:20.286150 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:20.286118    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ea92bdf-3a68-49be-80fe-b32ea7e220a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea92bdf-3a68-49be-80fe-b32ea7e220a2-maas-api-openshift-ca\") pod \"0ea92bdf-3a68-49be-80fe-b32ea7e220a2\" (UID: \"0ea92bdf-3a68-49be-80fe-b32ea7e220a2\") "
Oct 31 03:53:20.286326 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:20.286177    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cb4f9a2-dfba-43a6-8d42-a68281d62d37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb4f9a2-dfba-43a6-8d42-a68281d62d37-maas-api-xks-ca\") pod \"0cb4f9a2-dfba-43a6-8d42-a68281d62d37\" (UID: \"0cb4f9a2-dfba-43a6-8d42-a68281d62d37\") "
Oct 31 03:53:20.286326 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:20.286263    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ea92bdf-3a68-49be-80fe-b32ea7e220a2-kube-api-access-lpxx8\" (UniqueName: \"kubernetes.io/projected/0ea92bdf-3a68-49be-80fe-b32ea7e220a2-kube-api-access-lpxx8\") pod \"0ea92bdf-3a68-49be-80fe-b32ea7e220a2\" (UID: \"0ea92bdf-3a68-49be-80fe-b32ea7e220a2\") "
Oct 31 03:53:20.286326 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:20.286300    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cb4f9a2-dfba-43a6-8d42-a68281d62d37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb4f9a2-dfba-43a6-8d42-a68281d62d37-maas-api-openshift-ca\") pod \"0cb4f9a2-dfba-43a6-8d42-a68281d62d37\" (UID: \"0cb4f9a2-dfba-43a6-8d42-a68281d62d37\") "
Oct 31 03:53:20.286504 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:20.286460    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ea92bdf-3a68-49be-80fe-b32ea7e220a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea92bdf-3a68-49be-80fe-b32ea7e220a2-maas-api-xks-ca\") pod \"0ea92bdf-3a68-49be-80fe-b32ea7e220a2\" (UID: \"0ea92bdf-3a68-49be-80fe-b32ea7e220a2\") "
Oct 31 03:53:20.286559 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:20.286508    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ea92bdf-3a68-49be-80fe-b32ea7e220a2-maas-api-openshift-ca" pod "0ea92bdf-3a68-49be-80fe-b32ea7e220a2" (UID: "0ea92bdf-3a68-49be-80fe-b32ea7e220a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:53:20.286746 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:20.286719    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cb4f9a2-dfba-43a6-8d42-a68281d62d37-kube-api-access-h7cvc\" (UniqueName: \"kubernetes.io/projected/0cb4f9a2-dfba-43a6-8d42-a68281d62d37-kube-api-access-h7cvc\") pod \"0cb4f9a2-dfba-43a6-8d42-a68281d62d37\" (UID: \"0cb4f9a2-dfba-43a6-8d42-a68281d62d37\") "
Oct 31 03:53:20.286897 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:20.286725    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb4f9a2-dfba-43a6-8d42-a68281d62d37-maas-api-openshift-ca" pod "0cb4f9a2-dfba-43a6-8d42-a68281d62d37" (UID: "0cb4f9a2-dfba-43a6-8d42-a68281d62d37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:53:20.287699 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:20.287672    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea92bdf-3a68-49be-80fe-b32ea7e220a2-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:53:20.287817 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:20.287703    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb4f9a2-dfba-43a6-8d42-a68281d62d37-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:53:20.288926 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:20.288898    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb4f9a2-dfba-43a6-8d42-a68281d62d37-maas-api-xks-ca" pod "0cb4f9a2-dfba-43a6-8d42-a68281d62d37" (UID: "0cb4f9a2-dfba-43a6-8d42-a68281d62d37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:53:20.289045 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:20.288959    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb4f9a2-dfba-43a6-8d42-a68281d62d37-kube-api-access-h7cvc" pod "0cb4f9a2-dfba-43a6-8d42-a68281d62d37" (UID: "0cb4f9a2-dfba-43a6-8d42-a68281d62d37"). InnerVolumeSpecName "kube-api-access-h7cvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:53:20.289045 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:20.289004    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ea92bdf-3a68-49be-80fe-b32ea7e220a2-maas-api-xks-ca" pod "0ea92bdf-3a68-49be-80fe-b32ea7e220a2" (UID: "0ea92bdf-3a68-49be-80fe-b32ea7e220a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:53:20.289371 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:20.289355    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea92bdf-3a68-49be-80fe-b32ea7e220a2-kube-api-access-lpxx8" pod "0ea92bdf-3a68-49be-80fe-b32ea7e220a2" (UID: "0ea92bdf-3a68-49be-80fe-b32ea7e220a2"). InnerVolumeSpecName "kube-api-access-lpxx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:53:20.389902 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:20.389810    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7cvc\" (UniqueName: \"kubernetes.io/projected/0cb4f9a2-dfba-43a6-8d42-a68281d62d37-kube-api-access-h7cvc\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:53:20.389902 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:20.389850    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb4f9a2-dfba-43a6-8d42-a68281d62d37-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:53:20.389902 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:20.389865    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpxx8\" (UniqueName: \"kubernetes.io/projected/0ea92bdf-3a68-49be-80fe-b32ea7e220a2-kube-api-access-lpxx8\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:53:20.390136 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:20.389916    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea92bdf-3a68-49be-80fe-b32ea7e220a2-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:53:21.065690 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:21.065655    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba367b2-kkk7d"
Oct 31 03:53:21.065690 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:21.065677    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba367b2-kkk7d" event={"ID":"0ea92bdf-3a68-49be-80fe-b32ea7e220a2","Type":"ContainerDied","Data":"12e237fe6d8e74a6de983e268d8cae6824fc030e7f76930923bc28e24ae8c389"}
Oct 31 03:53:21.065970 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:21.065732    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12e237fe6d8e74a6de983e268d8cae6824fc030e7f76930923bc28e24ae8c389"
Oct 31 03:53:21.067312 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:21.067291    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d114f70b-lcntl"
Oct 31 03:53:21.067436 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:21.067327    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d114f70b-lcntl" event={"ID":"0cb4f9a2-dfba-43a6-8d42-a68281d62d37","Type":"ContainerDied","Data":"ac27194d804d9c35dacf909f5f1a412bf30a4abffe295ab4f0964db5191bc714"}
Oct 31 03:53:21.067436 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:53:21.067367    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac27194d804d9c35dacf909f5f1a412bf30a4abffe295ab4f0964db5191bc714"
Oct 31 03:53:49.003621 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:53:49.003582    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:54:02.732964 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:02.732924    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-rcm2n"]
Oct 31 03:54:02.782140 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:02.782100    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-rcm2n"]
Oct 31 03:54:04.609857 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:04.609814    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c2f4dff-e3f0-4716-a107-aa20fb362160" path="/var/lib/kubelet/pods/5c2f4dff-e3f0-4716-a107-aa20fb362160/volumes"
Oct 31 03:54:13.163771 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.163729    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5"]
Oct 31 03:54:13.164784 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.164760    4447 cpu_manager.go:414] "containerMap: removing container" podUID="0ea92bdf-3a68-49be-80fe-b32ea7e220a2" containerName="revoke-keys"
Oct 31 03:54:13.164988 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.164970    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ea92bdf-3a68-49be-80fe-b32ea7e220a2" containerName="revoke-keys"
Oct 31 03:54:13.165130 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.165117    4447 cpu_manager.go:414] "containerMap: removing container" podUID="0cb4f9a2-dfba-43a6-8d42-a68281d62d37" containerName="revoke-keys"
Oct 31 03:54:13.165237 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.165216    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cb4f9a2-dfba-43a6-8d42-a68281d62d37" containerName="revoke-keys"
Oct 31 03:54:13.165467 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.165449    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ea92bdf-3a68-49be-80fe-b32ea7e220a2" containerName="revoke-keys"
Oct 31 03:54:13.165598 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.165586    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cb4f9a2-dfba-43a6-8d42-a68281d62d37" containerName="revoke-keys"
Oct 31 03:54:13.191045 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.191013    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5"
Oct 31 03:54:13.251907 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.231699    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5"]
Oct 31 03:54:13.273190 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.273157    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1180d818-dockercfg-8hljd\""
Oct 31 03:54:13.280934 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.274103    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1180d818\""
Oct 31 03:54:13.314083 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.312216    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac96790-89ec-43aa-8949-7a8bc510e676-plugins-config-volume\") pod \"payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5\" (UID: \"dac96790-89ec-43aa-8949-7a8bc510e676\") " pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5"
Oct 31 03:54:13.314694 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.314628    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c978d\" (UniqueName: \"kubernetes.io/projected/dac96790-89ec-43aa-8949-7a8bc510e676-kube-api-access-c978d\") pod \"payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5\" (UID: \"dac96790-89ec-43aa-8949-7a8bc510e676\") " pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5"
Oct 31 03:54:13.355088 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.355057    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg"]
Oct 31 03:54:13.359967 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.359937    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg"
Oct 31 03:54:13.416375 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.416298    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkdqg\" (UniqueName: \"kubernetes.io/projected/aebb4eab-1d36-4aad-bcc4-e62ec26db1d4-kube-api-access-xkdqg\") pod \"payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg\" (UID: \"aebb4eab-1d36-4aad-bcc4-e62ec26db1d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg"
Oct 31 03:54:13.416558 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.416377    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c978d\" (UniqueName: \"kubernetes.io/projected/dac96790-89ec-43aa-8949-7a8bc510e676-kube-api-access-c978d\") pod \"payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5\" (UID: \"dac96790-89ec-43aa-8949-7a8bc510e676\") " pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5"
Oct 31 03:54:13.416558 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.416374    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg"]
Oct 31 03:54:13.416558 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.416505    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac96790-89ec-43aa-8949-7a8bc510e676-plugins-config-volume\") pod \"payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5\" (UID: \"dac96790-89ec-43aa-8949-7a8bc510e676\") " pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5"
Oct 31 03:54:13.416777 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.416627    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aebb4eab-1d36-4aad-bcc4-e62ec26db1d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg\" (UID: \"aebb4eab-1d36-4aad-bcc4-e62ec26db1d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg"
Oct 31 03:54:13.417416 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.417388    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac96790-89ec-43aa-8949-7a8bc510e676-plugins-config-volume\") pod \"payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5\" (UID: \"dac96790-89ec-43aa-8949-7a8bc510e676\") " pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5"
Oct 31 03:54:13.477699 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.477661    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c978d\" (UniqueName: \"kubernetes.io/projected/dac96790-89ec-43aa-8949-7a8bc510e676-kube-api-access-c978d\") pod \"payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5\" (UID: \"dac96790-89ec-43aa-8949-7a8bc510e676\") " pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5"
Oct 31 03:54:13.518001 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.517971    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkdqg\" (UniqueName: \"kubernetes.io/projected/aebb4eab-1d36-4aad-bcc4-e62ec26db1d4-kube-api-access-xkdqg\") pod \"payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg\" (UID: \"aebb4eab-1d36-4aad-bcc4-e62ec26db1d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg"
Oct 31 03:54:13.518177 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.518120    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aebb4eab-1d36-4aad-bcc4-e62ec26db1d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg\" (UID: \"aebb4eab-1d36-4aad-bcc4-e62ec26db1d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg"
Oct 31 03:54:13.518767 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.518748    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aebb4eab-1d36-4aad-bcc4-e62ec26db1d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg\" (UID: \"aebb4eab-1d36-4aad-bcc4-e62ec26db1d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg"
Oct 31 03:54:13.544999 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.544967    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5"
Oct 31 03:54:13.579690 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.579660    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkdqg\" (UniqueName: \"kubernetes.io/projected/aebb4eab-1d36-4aad-bcc4-e62ec26db1d4-kube-api-access-xkdqg\") pod \"payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg\" (UID: \"aebb4eab-1d36-4aad-bcc4-e62ec26db1d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg"
Oct 31 03:54:13.675180 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.674527    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg"
Oct 31 03:54:13.929825 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:13.929792    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5"]
Oct 31 03:54:13.930035 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:54:13.930013    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac96790_89ec_43aa_8949_7a8bc510e676.slice/crio-8c9fd1116b719c013753cc8e1530646bcd46d8d31d433e550c6f01859fefab34 WatchSource:0}: Error finding container 8c9fd1116b719c013753cc8e1530646bcd46d8d31d433e550c6f01859fefab34: Status 404 returned error can't find the container with id 8c9fd1116b719c013753cc8e1530646bcd46d8d31d433e550c6f01859fefab34
Oct 31 03:54:14.055947 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:54:14.055921    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaebb4eab_1d36_4aad_bcc4_e62ec26db1d4.slice/crio-e5a44259fe35a097faaf6a5ceb0b222bc3ef0681753332caa6c759a0a0084ca8 WatchSource:0}: Error finding container e5a44259fe35a097faaf6a5ceb0b222bc3ef0681753332caa6c759a0a0084ca8: Status 404 returned error can't find the container with id e5a44259fe35a097faaf6a5ceb0b222bc3ef0681753332caa6c759a0a0084ca8
Oct 31 03:54:14.056431 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:14.056404    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg"]
Oct 31 03:54:14.574629 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:14.574593    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5" event={"ID":"dac96790-89ec-43aa-8949-7a8bc510e676","Type":"ContainerStarted","Data":"8152409ed09e0cb8dac3d0ca5237eb2812da56b1f182070b5b0fbf565f38bc73"}
Oct 31 03:54:14.575096 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:14.574658    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5" event={"ID":"dac96790-89ec-43aa-8949-7a8bc510e676","Type":"ContainerStarted","Data":"8c9fd1116b719c013753cc8e1530646bcd46d8d31d433e550c6f01859fefab34"}
Oct 31 03:54:14.575096 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:14.574685    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5"
Oct 31 03:54:14.576352 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:14.576325    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg" event={"ID":"aebb4eab-1d36-4aad-bcc4-e62ec26db1d4","Type":"ContainerStarted","Data":"ac1b9e64dc4fb83baebeae7b222377b6def0144ad691a3b2a743158f7eb01dfc"}
Oct 31 03:54:14.576466 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:14.576377    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg" event={"ID":"aebb4eab-1d36-4aad-bcc4-e62ec26db1d4","Type":"ContainerStarted","Data":"e5a44259fe35a097faaf6a5ceb0b222bc3ef0681753332caa6c759a0a0084ca8"}
Oct 31 03:54:14.576466 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:14.576415    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg"
Oct 31 03:54:14.694722 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:14.694663    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5" podStartSLOduration=1.694647842 podStartE2EDuration="1.694647842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:14.692403127 +0000 UTC m=+2080.203385510" watchObservedRunningTime="2026-10-31 03:54:14.694647842 +0000 UTC m=+2080.205630218"
Oct 31 03:54:14.824337 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:14.822948    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg" podStartSLOduration=1.822927161 podStartE2EDuration="1.822927161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:14.819181674 +0000 UTC m=+2080.330164052" watchObservedRunningTime="2026-10-31 03:54:14.822927161 +0000 UTC m=+2080.333909539"
Oct 31 03:54:15.648283 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:15.648248    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"]
Oct 31 03:54:15.648688 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:15.648476    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4" podUID="e3c66081-9e3d-4097-8ba5-0624bcfcdc57" containerName="istio-proxy" containerID="cri-o://b080e3ad729e8cd21ff47475185d585c0f3d35734542445ba17d0c6ed884a695" gracePeriod=30
Oct 31 03:54:18.666763 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:18.666728    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf"]
Oct 31 03:54:18.671894 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:18.671861    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf"
Oct 31 03:54:18.694865 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:18.694828    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aab3ee5-2dbc-4227-9267-4f9c05857305-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf\" (UID: \"4aab3ee5-2dbc-4227-9267-4f9c05857305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf"
Oct 31 03:54:18.695062 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:18.695014    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkwv9\" (UniqueName: \"kubernetes.io/projected/4aab3ee5-2dbc-4227-9267-4f9c05857305-kube-api-access-bkwv9\") pod \"maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf\" (UID: \"4aab3ee5-2dbc-4227-9267-4f9c05857305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf"
Oct 31 03:54:18.695276 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:18.695258    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aab3ee5-2dbc-4227-9267-4f9c05857305-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf\" (UID: \"4aab3ee5-2dbc-4227-9267-4f9c05857305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf"
Oct 31 03:54:18.730006 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:18.729974    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf"]
Oct 31 03:54:18.774148 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:18.774113    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:54:18.774308 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:18.774157    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 03:54:18.797142 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:18.797110    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aab3ee5-2dbc-4227-9267-4f9c05857305-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf\" (UID: \"4aab3ee5-2dbc-4227-9267-4f9c05857305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf"
Oct 31 03:54:18.797292 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:18.797181    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aab3ee5-2dbc-4227-9267-4f9c05857305-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf\" (UID: \"4aab3ee5-2dbc-4227-9267-4f9c05857305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf"
Oct 31 03:54:18.797292 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:18.797257    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkwv9\" (UniqueName: \"kubernetes.io/projected/4aab3ee5-2dbc-4227-9267-4f9c05857305-kube-api-access-bkwv9\") pod \"maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf\" (UID: \"4aab3ee5-2dbc-4227-9267-4f9c05857305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf"
Oct 31 03:54:18.797848 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:18.797827    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aab3ee5-2dbc-4227-9267-4f9c05857305-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf\" (UID: \"4aab3ee5-2dbc-4227-9267-4f9c05857305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf"
Oct 31 03:54:18.799702 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:18.799681    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aab3ee5-2dbc-4227-9267-4f9c05857305-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf\" (UID: \"4aab3ee5-2dbc-4227-9267-4f9c05857305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf"
Oct 31 03:54:18.858366 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:18.858330    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkwv9\" (UniqueName: \"kubernetes.io/projected/4aab3ee5-2dbc-4227-9267-4f9c05857305-kube-api-access-bkwv9\") pod \"maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf\" (UID: \"4aab3ee5-2dbc-4227-9267-4f9c05857305\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf"
Oct 31 03:54:18.987361 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:18.987321    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf"
Oct 31 03:54:19.005051 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:54:19.005017    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:54:19.366589 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:54:19.366552    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aab3ee5_2dbc_4227_9267_4f9c05857305.slice/crio-159f5d16a21551d9aa8791271f3c4c6ad252e6b12e63c61cb82b5ad710e2d949 WatchSource:0}: Error finding container 159f5d16a21551d9aa8791271f3c4c6ad252e6b12e63c61cb82b5ad710e2d949: Status 404 returned error can't find the container with id 159f5d16a21551d9aa8791271f3c4c6ad252e6b12e63c61cb82b5ad710e2d949
Oct 31 03:54:19.368116 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:19.368087    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf"]
Oct 31 03:54:19.618304 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:19.618224    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf" event={"ID":"4aab3ee5-2dbc-4227-9267-4f9c05857305","Type":"ContainerStarted","Data":"159f5d16a21551d9aa8791271f3c4c6ad252e6b12e63c61cb82b5ad710e2d949"}
Oct 31 03:54:20.627932 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:20.627826    4447 generic.go:356] "Generic (PLEG): container finished" podID="4aab3ee5-2dbc-4227-9267-4f9c05857305" containerID="f43e12383984fdb0bb785e3f050af01be2fab1ecbd8ffe695afd6c21f4a94222" exitCode=0
Oct 31 03:54:20.627932 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:20.627916    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf" event={"ID":"4aab3ee5-2dbc-4227-9267-4f9c05857305","Type":"ContainerDied","Data":"f43e12383984fdb0bb785e3f050af01be2fab1ecbd8ffe695afd6c21f4a94222"}
Oct 31 03:54:20.930592 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:20.930566    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:54:21.029614 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.029584    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-data\") pod \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") "
Oct 31 03:54:21.029614 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.029620    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-podinfo\") pod \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") "
Oct 31 03:54:21.029868 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.029683    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-token\" (UniqueName: \"kubernetes.io/projected/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-token\") pod \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") "
Oct 31 03:54:21.029868 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.029823    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-kube-api-access-p24jk\" (UniqueName: \"kubernetes.io/projected/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-kube-api-access-p24jk\") pod \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") "
Oct 31 03:54:21.029868 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.029854    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-data" pod "e3c66081-9e3d-4097-8ba5-0624bcfcdc57" (UID: "e3c66081-9e3d-4097-8ba5-0624bcfcdc57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:21.030083 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.029870    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-workload-socket\") pod \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") "
Oct 31 03:54:21.030083 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.029989    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-credential-socket\") pod \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") "
Oct 31 03:54:21.030083 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.030023    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-envoy\") pod \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") "
Oct 31 03:54:21.030249 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.030083    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istiod-ca-cert\") pod \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") "
Oct 31 03:54:21.030249 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.030113    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-workload-certs\") pod \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\" (UID: \"e3c66081-9e3d-4097-8ba5-0624bcfcdc57\") "
Oct 31 03:54:21.030249 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.030124    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-workload-socket" pod "e3c66081-9e3d-4097-8ba5-0624bcfcdc57" (UID: "e3c66081-9e3d-4097-8ba5-0624bcfcdc57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:21.030388 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.030270    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-credential-socket" pod "e3c66081-9e3d-4097-8ba5-0624bcfcdc57" (UID: "e3c66081-9e3d-4097-8ba5-0624bcfcdc57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:21.030489 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.030466    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istiod-ca-cert" pod "e3c66081-9e3d-4097-8ba5-0624bcfcdc57" (UID: "e3c66081-9e3d-4097-8ba5-0624bcfcdc57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:54:21.030712 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.030479    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-workload-certs" pod "e3c66081-9e3d-4097-8ba5-0624bcfcdc57" (UID: "e3c66081-9e3d-4097-8ba5-0624bcfcdc57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:21.030769 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.030745    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:21.030769 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.030758    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:21.030769 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.030767    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:21.030962 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.030776    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:21.030962 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.030784    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:21.032219 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.032194    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-podinfo" pod "e3c66081-9e3d-4097-8ba5-0624bcfcdc57" (UID: "e3c66081-9e3d-4097-8ba5-0624bcfcdc57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:54:21.032387 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.032367    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-kube-api-access-p24jk" pod "e3c66081-9e3d-4097-8ba5-0624bcfcdc57" (UID: "e3c66081-9e3d-4097-8ba5-0624bcfcdc57"). InnerVolumeSpecName "kube-api-access-p24jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:21.032457 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.032408    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-token" pod "e3c66081-9e3d-4097-8ba5-0624bcfcdc57" (UID: "e3c66081-9e3d-4097-8ba5-0624bcfcdc57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:21.032872 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.032824    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-envoy" pod "e3c66081-9e3d-4097-8ba5-0624bcfcdc57" (UID: "e3c66081-9e3d-4097-8ba5-0624bcfcdc57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:21.132911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.132181    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:21.132911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.132222    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p24jk\" (UniqueName: \"kubernetes.io/projected/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-kube-api-access-p24jk\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:21.132911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.132237    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:21.132911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.132250    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c66081-9e3d-4097-8ba5-0624bcfcdc57-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:21.637338 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.637301    4447 generic.go:356] "Generic (PLEG): container finished" podID="e3c66081-9e3d-4097-8ba5-0624bcfcdc57" containerID="b080e3ad729e8cd21ff47475185d585c0f3d35734542445ba17d0c6ed884a695" exitCode=0
Oct 31 03:54:21.637798 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.637378    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"
Oct 31 03:54:21.637798 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.637397    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4" event={"ID":"e3c66081-9e3d-4097-8ba5-0624bcfcdc57","Type":"ContainerDied","Data":"b080e3ad729e8cd21ff47475185d585c0f3d35734542445ba17d0c6ed884a695"}
Oct 31 03:54:21.637798 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.637459    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4" event={"ID":"e3c66081-9e3d-4097-8ba5-0624bcfcdc57","Type":"ContainerDied","Data":"cfdbf47becf96e1da8f70910cbc7b3432a755c59f301d4b6b335edeba9191f93"}
Oct 31 03:54:21.637798 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.637489    4447 scope.go:122] "RemoveContainer" containerID="b080e3ad729e8cd21ff47475185d585c0f3d35734542445ba17d0c6ed884a695"
Oct 31 03:54:21.672799 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.672757    4447 scope.go:122] "RemoveContainer" containerID="b080e3ad729e8cd21ff47475185d585c0f3d35734542445ba17d0c6ed884a695"
Oct 31 03:54:21.673747 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:54:21.673694    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b080e3ad729e8cd21ff47475185d585c0f3d35734542445ba17d0c6ed884a695\": container with ID starting with b080e3ad729e8cd21ff47475185d585c0f3d35734542445ba17d0c6ed884a695 not found: ID does not exist" containerID="b080e3ad729e8cd21ff47475185d585c0f3d35734542445ba17d0c6ed884a695"
Oct 31 03:54:21.673873 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.673766    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b080e3ad729e8cd21ff47475185d585c0f3d35734542445ba17d0c6ed884a695"} err="failed to get container status \"b080e3ad729e8cd21ff47475185d585c0f3d35734542445ba17d0c6ed884a695\": rpc error: code = NotFound desc = could not find container \"b080e3ad729e8cd21ff47475185d585c0f3d35734542445ba17d0c6ed884a695\": container with ID starting with b080e3ad729e8cd21ff47475185d585c0f3d35734542445ba17d0c6ed884a695 not found: ID does not exist"
Oct 31 03:54:21.857497 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.856761    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"]
Oct 31 03:54:21.900363 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.900276    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-b9jd4"]
Oct 31 03:54:21.977230 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:21.977204    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf"
Oct 31 03:54:22.095825 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:22.095783    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aab3ee5-2dbc-4227-9267-4f9c05857305-kube-api-access-bkwv9\" (UniqueName: \"kubernetes.io/projected/4aab3ee5-2dbc-4227-9267-4f9c05857305-kube-api-access-bkwv9\") pod \"4aab3ee5-2dbc-4227-9267-4f9c05857305\" (UID: \"4aab3ee5-2dbc-4227-9267-4f9c05857305\") "
Oct 31 03:54:22.096138 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:22.095880    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aab3ee5-2dbc-4227-9267-4f9c05857305-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aab3ee5-2dbc-4227-9267-4f9c05857305-maas-api-openshift-ca\") pod \"4aab3ee5-2dbc-4227-9267-4f9c05857305\" (UID: \"4aab3ee5-2dbc-4227-9267-4f9c05857305\") "
Oct 31 03:54:22.096268 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:22.096157    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aab3ee5-2dbc-4227-9267-4f9c05857305-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aab3ee5-2dbc-4227-9267-4f9c05857305-maas-api-xks-ca\") pod \"4aab3ee5-2dbc-4227-9267-4f9c05857305\" (UID: \"4aab3ee5-2dbc-4227-9267-4f9c05857305\") "
Oct 31 03:54:22.096999 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:22.096960    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aab3ee5-2dbc-4227-9267-4f9c05857305-maas-api-openshift-ca" pod "4aab3ee5-2dbc-4227-9267-4f9c05857305" (UID: "4aab3ee5-2dbc-4227-9267-4f9c05857305"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:54:22.099565 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:22.099544    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aab3ee5-2dbc-4227-9267-4f9c05857305-maas-api-xks-ca" pod "4aab3ee5-2dbc-4227-9267-4f9c05857305" (UID: "4aab3ee5-2dbc-4227-9267-4f9c05857305"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:54:22.099869 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:22.099843    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aab3ee5-2dbc-4227-9267-4f9c05857305-kube-api-access-bkwv9" pod "4aab3ee5-2dbc-4227-9267-4f9c05857305" (UID: "4aab3ee5-2dbc-4227-9267-4f9c05857305"). InnerVolumeSpecName "kube-api-access-bkwv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:22.197854 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:22.197770    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aab3ee5-2dbc-4227-9267-4f9c05857305-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:22.197854 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:22.197798    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkwv9\" (UniqueName: \"kubernetes.io/projected/4aab3ee5-2dbc-4227-9267-4f9c05857305-kube-api-access-bkwv9\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:22.197854 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:22.197809    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aab3ee5-2dbc-4227-9267-4f9c05857305-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:22.612336 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:22.612299    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3c66081-9e3d-4097-8ba5-0624bcfcdc57" path="/var/lib/kubelet/pods/e3c66081-9e3d-4097-8ba5-0624bcfcdc57/volumes"
Oct 31 03:54:22.647360 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:22.647331    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf"
Oct 31 03:54:22.647834 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:22.647363    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf" event={"ID":"4aab3ee5-2dbc-4227-9267-4f9c05857305","Type":"ContainerDied","Data":"159f5d16a21551d9aa8791271f3c4c6ad252e6b12e63c61cb82b5ad710e2d949"}
Oct 31 03:54:22.647834 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:22.647407    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="159f5d16a21551d9aa8791271f3c4c6ad252e6b12e63c61cb82b5ad710e2d949"
Oct 31 03:54:25.587637 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:25.587606    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5"
Oct 31 03:54:25.588156 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:25.587819    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg"
Oct 31 03:54:26.858513 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:26.858475    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5190882d-fhcqv"]
Oct 31 03:54:26.859504 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:26.859485    4447 cpu_manager.go:414] "containerMap: removing container" podUID="4aab3ee5-2dbc-4227-9267-4f9c05857305" containerName="revoke-keys"
Oct 31 03:54:26.859560 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:26.859507    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aab3ee5-2dbc-4227-9267-4f9c05857305" containerName="revoke-keys"
Oct 31 03:54:26.859560 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:26.859556    4447 cpu_manager.go:414] "containerMap: removing container" podUID="e3c66081-9e3d-4097-8ba5-0624bcfcdc57" containerName="istio-proxy"
Oct 31 03:54:26.859625 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:26.859561    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c66081-9e3d-4097-8ba5-0624bcfcdc57" containerName="istio-proxy"
Oct 31 03:54:26.859688 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:26.859679    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c66081-9e3d-4097-8ba5-0624bcfcdc57" containerName="istio-proxy"
Oct 31 03:54:26.859724 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:26.859694    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aab3ee5-2dbc-4227-9267-4f9c05857305" containerName="revoke-keys"
Oct 31 03:54:26.863803 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:26.863785    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5190882d-fhcqv"
Oct 31 03:54:26.871667 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:26.871638    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d763a0-757d-4e92-afbb-25c2a839ef0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5190882d-fhcqv\" (UID: \"f1d763a0-757d-4e92-afbb-25c2a839ef0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5190882d-fhcqv"
Oct 31 03:54:26.871798 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:26.871701    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d763a0-757d-4e92-afbb-25c2a839ef0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5190882d-fhcqv\" (UID: \"f1d763a0-757d-4e92-afbb-25c2a839ef0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5190882d-fhcqv"
Oct 31 03:54:26.871868 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:26.871849    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcqbl\" (UniqueName: \"kubernetes.io/projected/f1d763a0-757d-4e92-afbb-25c2a839ef0b-kube-api-access-gcqbl\") pod \"maas-api-revoke-sub-5190882d-fhcqv\" (UID: \"f1d763a0-757d-4e92-afbb-25c2a839ef0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5190882d-fhcqv"
Oct 31 03:54:26.916865 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:26.916833    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:54:26.921612 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:26.921587    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5190882d-fhcqv"]
Oct 31 03:54:26.966141 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:26.966110    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 03:54:26.972914 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:26.972875    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d763a0-757d-4e92-afbb-25c2a839ef0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5190882d-fhcqv\" (UID: \"f1d763a0-757d-4e92-afbb-25c2a839ef0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5190882d-fhcqv"
Oct 31 03:54:26.973001 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:26.972935    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d763a0-757d-4e92-afbb-25c2a839ef0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5190882d-fhcqv\" (UID: \"f1d763a0-757d-4e92-afbb-25c2a839ef0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5190882d-fhcqv"
Oct 31 03:54:26.973048 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:26.973004    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcqbl\" (UniqueName: \"kubernetes.io/projected/f1d763a0-757d-4e92-afbb-25c2a839ef0b-kube-api-access-gcqbl\") pod \"maas-api-revoke-sub-5190882d-fhcqv\" (UID: \"f1d763a0-757d-4e92-afbb-25c2a839ef0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5190882d-fhcqv"
Oct 31 03:54:26.973489 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:26.973466    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d763a0-757d-4e92-afbb-25c2a839ef0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5190882d-fhcqv\" (UID: \"f1d763a0-757d-4e92-afbb-25c2a839ef0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5190882d-fhcqv"
Oct 31 03:54:26.975422 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:26.975394    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d763a0-757d-4e92-afbb-25c2a839ef0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5190882d-fhcqv\" (UID: \"f1d763a0-757d-4e92-afbb-25c2a839ef0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5190882d-fhcqv"
Oct 31 03:54:27.034830 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:27.034801    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcqbl\" (UniqueName: \"kubernetes.io/projected/f1d763a0-757d-4e92-afbb-25c2a839ef0b-kube-api-access-gcqbl\") pod \"maas-api-revoke-sub-5190882d-fhcqv\" (UID: \"f1d763a0-757d-4e92-afbb-25c2a839ef0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5190882d-fhcqv"
Oct 31 03:54:27.178030 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:27.177952    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5190882d-fhcqv"
Oct 31 03:54:27.557828 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:27.557802    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5190882d-fhcqv"]
Oct 31 03:54:27.557960 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:54:27.557798    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d763a0_757d_4e92_afbb_25c2a839ef0b.slice/crio-5b2c14f385771c17769aa97f2b94909b276a8dbe85873c04e6576f509f52c766 WatchSource:0}: Error finding container 5b2c14f385771c17769aa97f2b94909b276a8dbe85873c04e6576f509f52c766: Status 404 returned error can't find the container with id 5b2c14f385771c17769aa97f2b94909b276a8dbe85873c04e6576f509f52c766
Oct 31 03:54:27.691037 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:27.691008    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5190882d-fhcqv" event={"ID":"f1d763a0-757d-4e92-afbb-25c2a839ef0b","Type":"ContainerStarted","Data":"5b2c14f385771c17769aa97f2b94909b276a8dbe85873c04e6576f509f52c766"}
Oct 31 03:54:28.701208 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:28.701123    4447 generic.go:356] "Generic (PLEG): container finished" podID="f1d763a0-757d-4e92-afbb-25c2a839ef0b" containerID="e9c48ca3ef9116e1320fc9c0de398d17bcf70061bbe527d774f2ade99bb3aaf4" exitCode=0
Oct 31 03:54:28.701208 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:28.701183    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5190882d-fhcqv" event={"ID":"f1d763a0-757d-4e92-afbb-25c2a839ef0b","Type":"ContainerDied","Data":"e9c48ca3ef9116e1320fc9c0de398d17bcf70061bbe527d774f2ade99bb3aaf4"}
Oct 31 03:54:29.864279 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:29.864255    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5190882d-fhcqv"
Oct 31 03:54:29.912459 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:29.912425    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1d763a0-757d-4e92-afbb-25c2a839ef0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d763a0-757d-4e92-afbb-25c2a839ef0b-maas-api-xks-ca\") pod \"f1d763a0-757d-4e92-afbb-25c2a839ef0b\" (UID: \"f1d763a0-757d-4e92-afbb-25c2a839ef0b\") "
Oct 31 03:54:29.912630 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:29.912607    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d763a0-757d-4e92-afbb-25c2a839ef0b-kube-api-access-gcqbl\" (UniqueName: \"kubernetes.io/projected/f1d763a0-757d-4e92-afbb-25c2a839ef0b-kube-api-access-gcqbl\") pod \"f1d763a0-757d-4e92-afbb-25c2a839ef0b\" (UID: \"f1d763a0-757d-4e92-afbb-25c2a839ef0b\") "
Oct 31 03:54:29.912674 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:29.912636    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1d763a0-757d-4e92-afbb-25c2a839ef0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d763a0-757d-4e92-afbb-25c2a839ef0b-maas-api-openshift-ca\") pod \"f1d763a0-757d-4e92-afbb-25c2a839ef0b\" (UID: \"f1d763a0-757d-4e92-afbb-25c2a839ef0b\") "
Oct 31 03:54:29.913083 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:29.913044    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d763a0-757d-4e92-afbb-25c2a839ef0b-maas-api-openshift-ca" pod "f1d763a0-757d-4e92-afbb-25c2a839ef0b" (UID: "f1d763a0-757d-4e92-afbb-25c2a839ef0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:54:29.914694 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:29.914667    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d763a0-757d-4e92-afbb-25c2a839ef0b-maas-api-xks-ca" pod "f1d763a0-757d-4e92-afbb-25c2a839ef0b" (UID: "f1d763a0-757d-4e92-afbb-25c2a839ef0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:54:29.914829 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:29.914812    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d763a0-757d-4e92-afbb-25c2a839ef0b-kube-api-access-gcqbl" pod "f1d763a0-757d-4e92-afbb-25c2a839ef0b" (UID: "f1d763a0-757d-4e92-afbb-25c2a839ef0b"). InnerVolumeSpecName "kube-api-access-gcqbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:30.016733 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:30.016637    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcqbl\" (UniqueName: \"kubernetes.io/projected/f1d763a0-757d-4e92-afbb-25c2a839ef0b-kube-api-access-gcqbl\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:30.016733 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:30.016684    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d763a0-757d-4e92-afbb-25c2a839ef0b-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:30.016733 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:30.016701    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d763a0-757d-4e92-afbb-25c2a839ef0b-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:30.717989 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:30.717961    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5190882d-fhcqv"
Oct 31 03:54:30.718177 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:30.717962    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5190882d-fhcqv" event={"ID":"f1d763a0-757d-4e92-afbb-25c2a839ef0b","Type":"ContainerDied","Data":"5b2c14f385771c17769aa97f2b94909b276a8dbe85873c04e6576f509f52c766"}
Oct 31 03:54:30.718177 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:30.718080    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b2c14f385771c17769aa97f2b94909b276a8dbe85873c04e6576f509f52c766"
Oct 31 03:54:33.050055 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:33.050015    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"]
Oct 31 03:54:33.050526 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:33.050281    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v" podUID="5f030d4e-f850-4a06-a262-d2060e084bbf" containerName="istio-proxy" containerID="cri-o://f97383e94f5f715c98d147432d45f39b4d49bb03c2efe9896601d6999fcaf65e" gracePeriod=30
Oct 31 03:54:34.521636 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:34.521601    4447 patch_prober.go:29] interesting pod/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:54:34.522043 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:34.521681    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v" podUID="5f030d4e-f850-4a06-a262-d2060e084bbf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:54:34.964307 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:34.964281    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 03:54:34.983450 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:34.983425    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 03:54:38.328772 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.328710    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:54:38.457471 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.457431    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-workload-socket\") pod \"5f030d4e-f850-4a06-a262-d2060e084bbf\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") "
Oct 31 03:54:38.457657 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.457486    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f030d4e-f850-4a06-a262-d2060e084bbf-kube-api-access-d7cjw\" (UniqueName: \"kubernetes.io/projected/5f030d4e-f850-4a06-a262-d2060e084bbf-kube-api-access-d7cjw\") pod \"5f030d4e-f850-4a06-a262-d2060e084bbf\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") "
Oct 31 03:54:38.457657 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.457524    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-envoy\") pod \"5f030d4e-f850-4a06-a262-d2060e084bbf\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") "
Oct 31 03:54:38.457657 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.457544    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-token\" (UniqueName: \"kubernetes.io/projected/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-token\") pod \"5f030d4e-f850-4a06-a262-d2060e084bbf\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") "
Oct 31 03:54:38.457827 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.457759    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-data\") pod \"5f030d4e-f850-4a06-a262-d2060e084bbf\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") "
Oct 31 03:54:38.457827 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.457761    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-workload-socket" pod "5f030d4e-f850-4a06-a262-d2060e084bbf" (UID: "5f030d4e-f850-4a06-a262-d2060e084bbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:38.457977 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.457913    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-workload-certs\") pod \"5f030d4e-f850-4a06-a262-d2060e084bbf\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") "
Oct 31 03:54:38.457977 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.457946    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f030d4e-f850-4a06-a262-d2060e084bbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f030d4e-f850-4a06-a262-d2060e084bbf-istiod-ca-cert\") pod \"5f030d4e-f850-4a06-a262-d2060e084bbf\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") "
Oct 31 03:54:38.458072 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.458023    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-data" pod "5f030d4e-f850-4a06-a262-d2060e084bbf" (UID: "5f030d4e-f850-4a06-a262-d2060e084bbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:38.458126 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.458088    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-credential-socket\") pod \"5f030d4e-f850-4a06-a262-d2060e084bbf\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") "
Oct 31 03:54:38.458188 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.458122    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-workload-certs" pod "5f030d4e-f850-4a06-a262-d2060e084bbf" (UID: "5f030d4e-f850-4a06-a262-d2060e084bbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:38.458244 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.458180    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-podinfo\") pod \"5f030d4e-f850-4a06-a262-d2060e084bbf\" (UID: \"5f030d4e-f850-4a06-a262-d2060e084bbf\") "
Oct 31 03:54:38.458363 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.458342    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f030d4e-f850-4a06-a262-d2060e084bbf-istiod-ca-cert" pod "5f030d4e-f850-4a06-a262-d2060e084bbf" (UID: "5f030d4e-f850-4a06-a262-d2060e084bbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:54:38.458548 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.458526    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-credential-socket" pod "5f030d4e-f850-4a06-a262-d2060e084bbf" (UID: "5f030d4e-f850-4a06-a262-d2060e084bbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:38.459035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.459008    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:38.459144 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.459039    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:38.459144 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.459056    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f030d4e-f850-4a06-a262-d2060e084bbf-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:38.459144 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.459069    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:38.459144 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.459082    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:38.460098 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.460065    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-token" pod "5f030d4e-f850-4a06-a262-d2060e084bbf" (UID: "5f030d4e-f850-4a06-a262-d2060e084bbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:38.460197 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.460096    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-envoy" pod "5f030d4e-f850-4a06-a262-d2060e084bbf" (UID: "5f030d4e-f850-4a06-a262-d2060e084bbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:38.460197 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.460165    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f030d4e-f850-4a06-a262-d2060e084bbf-kube-api-access-d7cjw" pod "5f030d4e-f850-4a06-a262-d2060e084bbf" (UID: "5f030d4e-f850-4a06-a262-d2060e084bbf"). InnerVolumeSpecName "kube-api-access-d7cjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:38.460465 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.460446    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-podinfo" pod "5f030d4e-f850-4a06-a262-d2060e084bbf" (UID: "5f030d4e-f850-4a06-a262-d2060e084bbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:54:38.560605 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.560573    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:38.560605 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.560604    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7cjw\" (UniqueName: \"kubernetes.io/projected/5f030d4e-f850-4a06-a262-d2060e084bbf-kube-api-access-d7cjw\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:38.560790 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.560618    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:38.560790 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.560630    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f030d4e-f850-4a06-a262-d2060e084bbf-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:38.833238 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.833207    4447 generic.go:356] "Generic (PLEG): container finished" podID="5f030d4e-f850-4a06-a262-d2060e084bbf" containerID="f97383e94f5f715c98d147432d45f39b4d49bb03c2efe9896601d6999fcaf65e" exitCode=0
Oct 31 03:54:38.833419 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.833270    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"
Oct 31 03:54:38.833419 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.833291    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v" event={"ID":"5f030d4e-f850-4a06-a262-d2060e084bbf","Type":"ContainerDied","Data":"f97383e94f5f715c98d147432d45f39b4d49bb03c2efe9896601d6999fcaf65e"}
Oct 31 03:54:38.833419 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.833340    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v" event={"ID":"5f030d4e-f850-4a06-a262-d2060e084bbf","Type":"ContainerDied","Data":"2735a1eaba9bf040d6169f53579a7dbf85636ba1a41fe4791d3a25e8918daf45"}
Oct 31 03:54:38.833419 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.833361    4447 scope.go:122] "RemoveContainer" containerID="f97383e94f5f715c98d147432d45f39b4d49bb03c2efe9896601d6999fcaf65e"
Oct 31 03:54:38.844651 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.844633    4447 scope.go:122] "RemoveContainer" containerID="f97383e94f5f715c98d147432d45f39b4d49bb03c2efe9896601d6999fcaf65e"
Oct 31 03:54:38.844932 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:54:38.844915    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f97383e94f5f715c98d147432d45f39b4d49bb03c2efe9896601d6999fcaf65e\": container with ID starting with f97383e94f5f715c98d147432d45f39b4d49bb03c2efe9896601d6999fcaf65e not found: ID does not exist" containerID="f97383e94f5f715c98d147432d45f39b4d49bb03c2efe9896601d6999fcaf65e"
Oct 31 03:54:38.844987 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:38.844942    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f97383e94f5f715c98d147432d45f39b4d49bb03c2efe9896601d6999fcaf65e"} err="failed to get container status \"f97383e94f5f715c98d147432d45f39b4d49bb03c2efe9896601d6999fcaf65e\": rpc error: code = NotFound desc = could not find container \"f97383e94f5f715c98d147432d45f39b4d49bb03c2efe9896601d6999fcaf65e\": container with ID starting with f97383e94f5f715c98d147432d45f39b4d49bb03c2efe9896601d6999fcaf65e not found: ID does not exist"
Oct 31 03:54:39.001347 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:39.001311    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"]
Oct 31 03:54:39.050960 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:39.050922    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-dec4a0-openshift-default-7c778548ff-8qt9v"]
Oct 31 03:54:40.609611 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:40.609576    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f030d4e-f850-4a06-a262-d2060e084bbf" path="/var/lib/kubelet/pods/5f030d4e-f850-4a06-a262-d2060e084bbf/volumes"
Oct 31 03:54:42.611798 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:42.611766    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"]
Oct 31 03:54:42.612223 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:42.612178    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh" podUID="502e9c23-5ef9-4da4-93c4-5a5755d1f61e" containerName="maas-api" containerID="cri-o://2cecbc557288ecc00304c139afdf1b5ef662092258bb7902b532a531db883192" gracePeriod=30
Oct 31 03:54:42.901759 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:42.901679    4447 generic.go:356] "Generic (PLEG): container finished" podID="502e9c23-5ef9-4da4-93c4-5a5755d1f61e" containerID="2cecbc557288ecc00304c139afdf1b5ef662092258bb7902b532a531db883192" exitCode=0
Oct 31 03:54:42.901759 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:42.901747    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh" event={"ID":"502e9c23-5ef9-4da4-93c4-5a5755d1f61e","Type":"ContainerDied","Data":"2cecbc557288ecc00304c139afdf1b5ef662092258bb7902b532a531db883192"}
Oct 31 03:54:42.941710 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:42.941685    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"
Oct 31 03:54:43.038636 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:43.038601    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-maas-api-tls\") pod \"502e9c23-5ef9-4da4-93c4-5a5755d1f61e\" (UID: \"502e9c23-5ef9-4da4-93c4-5a5755d1f61e\") "
Oct 31 03:54:43.038848 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:43.038740    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-metrics-certs\") pod \"502e9c23-5ef9-4da4-93c4-5a5755d1f61e\" (UID: \"502e9c23-5ef9-4da4-93c4-5a5755d1f61e\") "
Oct 31 03:54:43.038848 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:43.038787    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-openshift-service-ca\") pod \"502e9c23-5ef9-4da4-93c4-5a5755d1f61e\" (UID: \"502e9c23-5ef9-4da4-93c4-5a5755d1f61e\") "
Oct 31 03:54:43.039025 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:43.038879    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-kube-api-access-4fpxj\" (UniqueName: \"kubernetes.io/projected/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-kube-api-access-4fpxj\") pod \"502e9c23-5ef9-4da4-93c4-5a5755d1f61e\" (UID: \"502e9c23-5ef9-4da4-93c4-5a5755d1f61e\") "
Oct 31 03:54:43.039467 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:43.039439    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-openshift-service-ca" pod "502e9c23-5ef9-4da4-93c4-5a5755d1f61e" (UID: "502e9c23-5ef9-4da4-93c4-5a5755d1f61e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:54:43.039868 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:43.039846    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-openshift-service-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:43.042606 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:43.042583    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-maas-api-tls" pod "502e9c23-5ef9-4da4-93c4-5a5755d1f61e" (UID: "502e9c23-5ef9-4da4-93c4-5a5755d1f61e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:54:43.045556 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:43.045533    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-metrics-certs" pod "502e9c23-5ef9-4da4-93c4-5a5755d1f61e" (UID: "502e9c23-5ef9-4da4-93c4-5a5755d1f61e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:54:43.045748 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:43.045718    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-kube-api-access-4fpxj" pod "502e9c23-5ef9-4da4-93c4-5a5755d1f61e" (UID: "502e9c23-5ef9-4da4-93c4-5a5755d1f61e"). InnerVolumeSpecName "kube-api-access-4fpxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:43.140814 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:43.140781    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-maas-api-tls\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:43.140814 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:43.140811    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-metrics-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:43.140814 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:43.140820    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fpxj\" (UniqueName: \"kubernetes.io/projected/502e9c23-5ef9-4da4-93c4-5a5755d1f61e-kube-api-access-4fpxj\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:43.911685 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:43.911652    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"
Oct 31 03:54:43.912196 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:43.911690    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh" event={"ID":"502e9c23-5ef9-4da4-93c4-5a5755d1f61e","Type":"ContainerDied","Data":"819b848198a7e82b4c29ea11288b75cede5dbbf5c72b02c4c311d041d9d1ddf8"}
Oct 31 03:54:43.912196 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:43.911751    4447 scope.go:122] "RemoveContainer" containerID="2cecbc557288ecc00304c139afdf1b5ef662092258bb7902b532a531db883192"
Oct 31 03:54:44.079450 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:44.079412    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"]
Oct 31 03:54:44.132447 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:44.132402    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e00ad3-996b96667-s5cqh"]
Oct 31 03:54:44.647656 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:44.647619    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="502e9c23-5ef9-4da4-93c4-5a5755d1f61e" path="/var/lib/kubelet/pods/502e9c23-5ef9-4da4-93c4-5a5755d1f61e/volumes"
Oct 31 03:54:44.789826 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:44.789784    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc"]
Oct 31 03:54:44.838980 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:44.838947    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e00ad3-p4pmc"]
Oct 31 03:54:46.609969 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:46.609869    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e9f85dd-3164-45e4-b907-e8c9bc3a2b71" path="/var/lib/kubelet/pods/6e9f85dd-3164-45e4-b907-e8c9bc3a2b71/volumes"
Oct 31 03:54:49.040919 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:54:49.040866    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:54:52.381576 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:52.381540    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"]
Oct 31 03:54:52.382187 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:52.381817    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm" podUID="bda8a9ea-d20f-4615-911a-46b42d93e700" containerName="istio-proxy" containerID="cri-o://3ee64b26b03331e6734c40aeccbba6e4de10aafcfb1df6f9d917b72c4e45416d" gracePeriod=30
Oct 31 03:54:55.152326 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:55.152287    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph"]
Oct 31 03:54:55.153082 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:55.153066    4447 cpu_manager.go:414] "containerMap: removing container" podUID="f1d763a0-757d-4e92-afbb-25c2a839ef0b" containerName="revoke-keys"
Oct 31 03:54:55.153140 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:55.153083    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d763a0-757d-4e92-afbb-25c2a839ef0b" containerName="revoke-keys"
Oct 31 03:54:55.153140 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:55.153108    4447 cpu_manager.go:414] "containerMap: removing container" podUID="5f030d4e-f850-4a06-a262-d2060e084bbf" containerName="istio-proxy"
Oct 31 03:54:55.153140 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:55.153113    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f030d4e-f850-4a06-a262-d2060e084bbf" containerName="istio-proxy"
Oct 31 03:54:55.153231 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:55.153144    4447 cpu_manager.go:414] "containerMap: removing container" podUID="502e9c23-5ef9-4da4-93c4-5a5755d1f61e" containerName="maas-api"
Oct 31 03:54:55.153231 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:55.153151    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="502e9c23-5ef9-4da4-93c4-5a5755d1f61e" containerName="maas-api"
Oct 31 03:54:55.153302 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:55.153251    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f030d4e-f850-4a06-a262-d2060e084bbf" containerName="istio-proxy"
Oct 31 03:54:55.153302 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:55.153264    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="502e9c23-5ef9-4da4-93c4-5a5755d1f61e" containerName="maas-api"
Oct 31 03:54:55.153302 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:55.153272    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d763a0-757d-4e92-afbb-25c2a839ef0b" containerName="revoke-keys"
Oct 31 03:54:55.158577 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:55.158561    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph"
Oct 31 03:54:55.214969 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:55.214934    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph"]
Oct 31 03:54:55.258827 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:55.258785    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:54:55.258827 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:55.258810    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 03:54:55.353970 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:55.353932    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwhdg\" (UniqueName: \"kubernetes.io/projected/5c8fd020-89d4-4f2b-93c1-115a0d20c7fd-kube-api-access-bwhdg\") pod \"maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph\" (UID: \"5c8fd020-89d4-4f2b-93c1-115a0d20c7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph"
Oct 31 03:54:55.354209 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:55.354083    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c8fd020-89d4-4f2b-93c1-115a0d20c7fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph\" (UID: \"5c8fd020-89d4-4f2b-93c1-115a0d20c7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph"
Oct 31 03:54:55.354209 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:55.354123    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c8fd020-89d4-4f2b-93c1-115a0d20c7fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph\" (UID: \"5c8fd020-89d4-4f2b-93c1-115a0d20c7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph"
Oct 31 03:54:55.455312 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:55.455282    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c8fd020-89d4-4f2b-93c1-115a0d20c7fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph\" (UID: \"5c8fd020-89d4-4f2b-93c1-115a0d20c7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph"
Oct 31 03:54:55.455477 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:55.455324    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c8fd020-89d4-4f2b-93c1-115a0d20c7fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph\" (UID: \"5c8fd020-89d4-4f2b-93c1-115a0d20c7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph"
Oct 31 03:54:55.455523 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:55.455499    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwhdg\" (UniqueName: \"kubernetes.io/projected/5c8fd020-89d4-4f2b-93c1-115a0d20c7fd-kube-api-access-bwhdg\") pod \"maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph\" (UID: \"5c8fd020-89d4-4f2b-93c1-115a0d20c7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph"
Oct 31 03:54:55.456197 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:55.456134    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c8fd020-89d4-4f2b-93c1-115a0d20c7fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph\" (UID: \"5c8fd020-89d4-4f2b-93c1-115a0d20c7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph"
Oct 31 03:54:55.458018 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:55.458000    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c8fd020-89d4-4f2b-93c1-115a0d20c7fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph\" (UID: \"5c8fd020-89d4-4f2b-93c1-115a0d20c7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph"
Oct 31 03:54:55.516609 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:55.516571    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwhdg\" (UniqueName: \"kubernetes.io/projected/5c8fd020-89d4-4f2b-93c1-115a0d20c7fd-kube-api-access-bwhdg\") pod \"maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph\" (UID: \"5c8fd020-89d4-4f2b-93c1-115a0d20c7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph"
Oct 31 03:54:55.773825 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:55.773743    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph"
Oct 31 03:54:56.158268 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:56.158231    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph"]
Oct 31 03:54:56.158747 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:54:56.158726    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8fd020_89d4_4f2b_93c1_115a0d20c7fd.slice/crio-3c8c9a3313a9b7d195cf5555c58ddfeb5e6de4c1feee08ed55f54eb7572714cf WatchSource:0}: Error finding container 3c8c9a3313a9b7d195cf5555c58ddfeb5e6de4c1feee08ed55f54eb7572714cf: Status 404 returned error can't find the container with id 3c8c9a3313a9b7d195cf5555c58ddfeb5e6de4c1feee08ed55f54eb7572714cf
Oct 31 03:54:57.038707 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.038674    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph" event={"ID":"5c8fd020-89d4-4f2b-93c1-115a0d20c7fd","Type":"ContainerStarted","Data":"ff24078326b5aedfab5d601e435fd5fedd23b0b2cb707a5bf3594ea21bec64e3"}
Oct 31 03:54:57.038972 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.038725    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph" event={"ID":"5c8fd020-89d4-4f2b-93c1-115a0d20c7fd","Type":"ContainerStarted","Data":"3c8c9a3313a9b7d195cf5555c58ddfeb5e6de4c1feee08ed55f54eb7572714cf"}
Oct 31 03:54:57.182133 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.182085    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph" podStartSLOduration=2.182071249 podStartE2EDuration="2.182071249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:57.154313051 +0000 UTC m=+2122.665295429" watchObservedRunningTime="2026-10-31 03:54:57.182071249 +0000 UTC m=+2122.693053625"
Oct 31 03:54:57.658392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.658335    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:54:57.679774 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.679749    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-data\") pod \"bda8a9ea-d20f-4615-911a-46b42d93e700\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") "
Oct 31 03:54:57.679949 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.679780    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bda8a9ea-d20f-4615-911a-46b42d93e700-kube-api-access-rqphh\" (UniqueName: \"kubernetes.io/projected/bda8a9ea-d20f-4615-911a-46b42d93e700-kube-api-access-rqphh\") pod \"bda8a9ea-d20f-4615-911a-46b42d93e700\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") "
Oct 31 03:54:57.679949 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.679813    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-podinfo\") pod \"bda8a9ea-d20f-4615-911a-46b42d93e700\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") "
Oct 31 03:54:57.679949 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.679916    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-envoy\") pod \"bda8a9ea-d20f-4615-911a-46b42d93e700\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") "
Oct 31 03:54:57.680106 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.679955    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-credential-socket\") pod \"bda8a9ea-d20f-4615-911a-46b42d93e700\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") "
Oct 31 03:54:57.680106 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.680007    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-token\" (UniqueName: \"kubernetes.io/projected/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-token\") pod \"bda8a9ea-d20f-4615-911a-46b42d93e700\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") "
Oct 31 03:54:57.680106 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.680044    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-workload-certs\") pod \"bda8a9ea-d20f-4615-911a-46b42d93e700\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") "
Oct 31 03:54:57.680106 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.680073    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bda8a9ea-d20f-4615-911a-46b42d93e700-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda8a9ea-d20f-4615-911a-46b42d93e700-istiod-ca-cert\") pod \"bda8a9ea-d20f-4615-911a-46b42d93e700\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") "
Oct 31 03:54:57.680106 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.680080    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-data" pod "bda8a9ea-d20f-4615-911a-46b42d93e700" (UID: "bda8a9ea-d20f-4615-911a-46b42d93e700"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:57.680279 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.680130    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-workload-socket\") pod \"bda8a9ea-d20f-4615-911a-46b42d93e700\" (UID: \"bda8a9ea-d20f-4615-911a-46b42d93e700\") "
Oct 31 03:54:57.680339 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.680253    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-credential-socket" pod "bda8a9ea-d20f-4615-911a-46b42d93e700" (UID: "bda8a9ea-d20f-4615-911a-46b42d93e700"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:57.680388 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.680345    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-workload-certs" pod "bda8a9ea-d20f-4615-911a-46b42d93e700" (UID: "bda8a9ea-d20f-4615-911a-46b42d93e700"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:57.680623 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.680563    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-workload-socket" pod "bda8a9ea-d20f-4615-911a-46b42d93e700" (UID: "bda8a9ea-d20f-4615-911a-46b42d93e700"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:57.680623 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.680639    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bda8a9ea-d20f-4615-911a-46b42d93e700-istiod-ca-cert" pod "bda8a9ea-d20f-4615-911a-46b42d93e700" (UID: "bda8a9ea-d20f-4615-911a-46b42d93e700"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:54:57.681508 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.681475    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:57.681508 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.681501    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:57.681706 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.681517    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda8a9ea-d20f-4615-911a-46b42d93e700-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:57.681706 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.681532    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:57.681706 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.681547    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:57.682610 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.682587    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-podinfo" pod "bda8a9ea-d20f-4615-911a-46b42d93e700" (UID: "bda8a9ea-d20f-4615-911a-46b42d93e700"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:54:57.682696 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.682620    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-token" pod "bda8a9ea-d20f-4615-911a-46b42d93e700" (UID: "bda8a9ea-d20f-4615-911a-46b42d93e700"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:57.682797 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.682776    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda8a9ea-d20f-4615-911a-46b42d93e700-kube-api-access-rqphh" pod "bda8a9ea-d20f-4615-911a-46b42d93e700" (UID: "bda8a9ea-d20f-4615-911a-46b42d93e700"). InnerVolumeSpecName "kube-api-access-rqphh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:57.682797 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.682792    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-envoy" pod "bda8a9ea-d20f-4615-911a-46b42d93e700" (UID: "bda8a9ea-d20f-4615-911a-46b42d93e700"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:57.783517 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.783479    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqphh\" (UniqueName: \"kubernetes.io/projected/bda8a9ea-d20f-4615-911a-46b42d93e700-kube-api-access-rqphh\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:57.783517 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.783516    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:57.783765 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.783530    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:57.783765 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:57.783545    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda8a9ea-d20f-4615-911a-46b42d93e700-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:58.047603 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:58.047556    4447 generic.go:356] "Generic (PLEG): container finished" podID="5c8fd020-89d4-4f2b-93c1-115a0d20c7fd" containerID="ff24078326b5aedfab5d601e435fd5fedd23b0b2cb707a5bf3594ea21bec64e3" exitCode=0
Oct 31 03:54:58.047811 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:58.047642    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph" event={"ID":"5c8fd020-89d4-4f2b-93c1-115a0d20c7fd","Type":"ContainerDied","Data":"ff24078326b5aedfab5d601e435fd5fedd23b0b2cb707a5bf3594ea21bec64e3"}
Oct 31 03:54:58.049105 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:58.049076    4447 generic.go:356] "Generic (PLEG): container finished" podID="bda8a9ea-d20f-4615-911a-46b42d93e700" containerID="3ee64b26b03331e6734c40aeccbba6e4de10aafcfb1df6f9d917b72c4e45416d" exitCode=0
Oct 31 03:54:58.049237 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:58.049116    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"
Oct 31 03:54:58.049237 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:58.049160    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm" event={"ID":"bda8a9ea-d20f-4615-911a-46b42d93e700","Type":"ContainerDied","Data":"3ee64b26b03331e6734c40aeccbba6e4de10aafcfb1df6f9d917b72c4e45416d"}
Oct 31 03:54:58.049237 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:58.049210    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm" event={"ID":"bda8a9ea-d20f-4615-911a-46b42d93e700","Type":"ContainerDied","Data":"2d336db7975013ee5cbf648ec644e79afe0179d1f1cb71d7b14ada0bdadb1b71"}
Oct 31 03:54:58.049356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:58.049237    4447 scope.go:122] "RemoveContainer" containerID="3ee64b26b03331e6734c40aeccbba6e4de10aafcfb1df6f9d917b72c4e45416d"
Oct 31 03:54:58.061413 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:58.061387    4447 scope.go:122] "RemoveContainer" containerID="3ee64b26b03331e6734c40aeccbba6e4de10aafcfb1df6f9d917b72c4e45416d"
Oct 31 03:54:58.061667 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:54:58.061648    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ee64b26b03331e6734c40aeccbba6e4de10aafcfb1df6f9d917b72c4e45416d\": container with ID starting with 3ee64b26b03331e6734c40aeccbba6e4de10aafcfb1df6f9d917b72c4e45416d not found: ID does not exist" containerID="3ee64b26b03331e6734c40aeccbba6e4de10aafcfb1df6f9d917b72c4e45416d"
Oct 31 03:54:58.061732 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:58.061675    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee64b26b03331e6734c40aeccbba6e4de10aafcfb1df6f9d917b72c4e45416d"} err="failed to get container status \"3ee64b26b03331e6734c40aeccbba6e4de10aafcfb1df6f9d917b72c4e45416d\": rpc error: code = NotFound desc = could not find container \"3ee64b26b03331e6734c40aeccbba6e4de10aafcfb1df6f9d917b72c4e45416d\": container with ID starting with 3ee64b26b03331e6734c40aeccbba6e4de10aafcfb1df6f9d917b72c4e45416d not found: ID does not exist"
Oct 31 03:54:58.278271 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:58.278235    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"]
Oct 31 03:54:58.282634 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:58.282608    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e00ad3-openshift-default-58867df88f-m4rgm"]
Oct 31 03:54:58.623859 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:58.623815    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bda8a9ea-d20f-4615-911a-46b42d93e700" path="/var/lib/kubelet/pods/bda8a9ea-d20f-4615-911a-46b42d93e700/volumes"
Oct 31 03:54:59.236121 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:59.236095    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph"
Oct 31 03:54:59.305021 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:59.304985    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c8fd020-89d4-4f2b-93c1-115a0d20c7fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c8fd020-89d4-4f2b-93c1-115a0d20c7fd-maas-api-xks-ca\") pod \"5c8fd020-89d4-4f2b-93c1-115a0d20c7fd\" (UID: \"5c8fd020-89d4-4f2b-93c1-115a0d20c7fd\") "
Oct 31 03:54:59.305411 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:59.305098    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c8fd020-89d4-4f2b-93c1-115a0d20c7fd-kube-api-access-bwhdg\" (UniqueName: \"kubernetes.io/projected/5c8fd020-89d4-4f2b-93c1-115a0d20c7fd-kube-api-access-bwhdg\") pod \"5c8fd020-89d4-4f2b-93c1-115a0d20c7fd\" (UID: \"5c8fd020-89d4-4f2b-93c1-115a0d20c7fd\") "
Oct 31 03:54:59.305411 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:59.305129    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c8fd020-89d4-4f2b-93c1-115a0d20c7fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c8fd020-89d4-4f2b-93c1-115a0d20c7fd-maas-api-openshift-ca\") pod \"5c8fd020-89d4-4f2b-93c1-115a0d20c7fd\" (UID: \"5c8fd020-89d4-4f2b-93c1-115a0d20c7fd\") "
Oct 31 03:54:59.305541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:59.305514    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c8fd020-89d4-4f2b-93c1-115a0d20c7fd-maas-api-openshift-ca" pod "5c8fd020-89d4-4f2b-93c1-115a0d20c7fd" (UID: "5c8fd020-89d4-4f2b-93c1-115a0d20c7fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:54:59.305835 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:59.305810    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c8fd020-89d4-4f2b-93c1-115a0d20c7fd-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:59.307212 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:59.307188    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c8fd020-89d4-4f2b-93c1-115a0d20c7fd-maas-api-xks-ca" pod "5c8fd020-89d4-4f2b-93c1-115a0d20c7fd" (UID: "5c8fd020-89d4-4f2b-93c1-115a0d20c7fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:54:59.307514 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:59.307493    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c8fd020-89d4-4f2b-93c1-115a0d20c7fd-kube-api-access-bwhdg" pod "5c8fd020-89d4-4f2b-93c1-115a0d20c7fd" (UID: "5c8fd020-89d4-4f2b-93c1-115a0d20c7fd"). InnerVolumeSpecName "kube-api-access-bwhdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:59.407353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:59.407316    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c8fd020-89d4-4f2b-93c1-115a0d20c7fd-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:54:59.407353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:54:59.407347    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwhdg\" (UniqueName: \"kubernetes.io/projected/5c8fd020-89d4-4f2b-93c1-115a0d20c7fd-kube-api-access-bwhdg\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:55:00.067110 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:00.067073    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph" event={"ID":"5c8fd020-89d4-4f2b-93c1-115a0d20c7fd","Type":"ContainerDied","Data":"3c8c9a3313a9b7d195cf5555c58ddfeb5e6de4c1feee08ed55f54eb7572714cf"}
Oct 31 03:55:00.067310 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:00.067134    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c8c9a3313a9b7d195cf5555c58ddfeb5e6de4c1feee08ed55f54eb7572714cf"
Oct 31 03:55:00.067310 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:00.067080    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph"
Oct 31 03:55:19.004165 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:55:19.004124    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:55:49.003959 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:55:49.003907    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:55:50.088539 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:50.088499    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84e130-nwtzj"]
Oct 31 03:55:50.089419 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:50.089397    4447 cpu_manager.go:414] "containerMap: removing container" podUID="bda8a9ea-d20f-4615-911a-46b42d93e700" containerName="istio-proxy"
Oct 31 03:55:50.089502 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:50.089421    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bda8a9ea-d20f-4615-911a-46b42d93e700" containerName="istio-proxy"
Oct 31 03:55:50.089502 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:50.089471    4447 cpu_manager.go:414] "containerMap: removing container" podUID="5c8fd020-89d4-4f2b-93c1-115a0d20c7fd" containerName="revoke-keys"
Oct 31 03:55:50.089502 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:50.089479    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c8fd020-89d4-4f2b-93c1-115a0d20c7fd" containerName="revoke-keys"
Oct 31 03:55:50.089650 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:50.089638    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="bda8a9ea-d20f-4615-911a-46b42d93e700" containerName="istio-proxy"
Oct 31 03:55:50.089684 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:50.089661    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c8fd020-89d4-4f2b-93c1-115a0d20c7fd" containerName="revoke-keys"
Oct 31 03:55:50.095355 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:50.095334    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84e130-nwtzj"
Oct 31 03:55:50.151588 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:50.151554    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84e130-nwtzj"]
Oct 31 03:55:50.165372 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:50.165336    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-84e130-nwtzj\" (UID: \"1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84e130-nwtzj"
Oct 31 03:55:50.165510 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:50.165485    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtwgf\" (UniqueName: \"kubernetes.io/projected/1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce-kube-api-access-gtwgf\") pod \"maas-api-revoke-keys-e2e-api-a-84e130-nwtzj\" (UID: \"1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84e130-nwtzj"
Oct 31 03:55:50.165649 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:50.165628    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-84e130-nwtzj\" (UID: \"1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84e130-nwtzj"
Oct 31 03:55:50.195987 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:50.195958    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:55:50.196154 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:50.196045    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 03:55:50.267284 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:50.267253    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-84e130-nwtzj\" (UID: \"1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84e130-nwtzj"
Oct 31 03:55:50.267461 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:50.267359    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtwgf\" (UniqueName: \"kubernetes.io/projected/1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce-kube-api-access-gtwgf\") pod \"maas-api-revoke-keys-e2e-api-a-84e130-nwtzj\" (UID: \"1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84e130-nwtzj"
Oct 31 03:55:50.267461 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:50.267418    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-84e130-nwtzj\" (UID: \"1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84e130-nwtzj"
Oct 31 03:55:50.268123 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:50.268044    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-84e130-nwtzj\" (UID: \"1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84e130-nwtzj"
Oct 31 03:55:50.269843 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:50.269820    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-84e130-nwtzj\" (UID: \"1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84e130-nwtzj"
Oct 31 03:55:50.329406 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:50.329374    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtwgf\" (UniqueName: \"kubernetes.io/projected/1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce-kube-api-access-gtwgf\") pod \"maas-api-revoke-keys-e2e-api-a-84e130-nwtzj\" (UID: \"1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84e130-nwtzj"
Oct 31 03:55:50.412198 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:50.412111    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84e130-nwtzj"
Oct 31 03:55:50.792069 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:55:50.792037    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd0924a_ecc1_4e38_b279_4a45a6fbc1ce.slice/crio-d030459e1b020977f6186fc37a7f4af8bca6045fc6a0a5a47a0a5a7d8c92d7d0 WatchSource:0}: Error finding container d030459e1b020977f6186fc37a7f4af8bca6045fc6a0a5a47a0a5a7d8c92d7d0: Status 404 returned error can't find the container with id d030459e1b020977f6186fc37a7f4af8bca6045fc6a0a5a47a0a5a7d8c92d7d0
Oct 31 03:55:50.792221 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:50.792200    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84e130-nwtzj"]
Oct 31 03:55:51.541191 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:51.541143    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84e130-nwtzj" event={"ID":"1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce","Type":"ContainerStarted","Data":"cb40d597a95cbd52cca69cb69c586e63ad00d306b717e8977a75e4151a8b7ae1"}
Oct 31 03:55:51.541665 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:51.541218    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84e130-nwtzj" event={"ID":"1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce","Type":"ContainerStarted","Data":"d030459e1b020977f6186fc37a7f4af8bca6045fc6a0a5a47a0a5a7d8c92d7d0"}
Oct 31 03:55:51.661468 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:51.661411    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84e130-nwtzj" podStartSLOduration=1.6613976369999999 podStartE2EDuration="1.661397637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:55:51.656728298 +0000 UTC m=+2177.167710674" watchObservedRunningTime="2026-10-31 03:55:51.661397637 +0000 UTC m=+2177.172380013"
Oct 31 03:55:52.551573 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:52.551541    4447 generic.go:356] "Generic (PLEG): container finished" podID="1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce" containerID="cb40d597a95cbd52cca69cb69c586e63ad00d306b717e8977a75e4151a8b7ae1" exitCode=0
Oct 31 03:55:52.552083 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:52.551632    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84e130-nwtzj" event={"ID":"1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce","Type":"ContainerDied","Data":"cb40d597a95cbd52cca69cb69c586e63ad00d306b717e8977a75e4151a8b7ae1"}
Oct 31 03:55:53.726110 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:53.726082    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84e130-nwtzj"
Oct 31 03:55:53.814021 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:53.813932    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce-maas-api-openshift-ca\") pod \"1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce\" (UID: \"1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce\") "
Oct 31 03:55:53.814021 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:53.814008    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce-kube-api-access-gtwgf\" (UniqueName: \"kubernetes.io/projected/1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce-kube-api-access-gtwgf\") pod \"1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce\" (UID: \"1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce\") "
Oct 31 03:55:53.814238 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:53.814109    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce-maas-api-xks-ca\") pod \"1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce\" (UID: \"1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce\") "
Oct 31 03:55:53.814312 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:53.814291    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce-maas-api-openshift-ca" pod "1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce" (UID: "1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:53.814731 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:53.814713    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:55:53.816363 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:53.816340    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce-maas-api-xks-ca" pod "1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce" (UID: "1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:55:53.816363 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:53.816357    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce-kube-api-access-gtwgf" pod "1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce" (UID: "1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce"). InnerVolumeSpecName "kube-api-access-gtwgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:53.916541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:53.916496    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtwgf\" (UniqueName: \"kubernetes.io/projected/1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce-kube-api-access-gtwgf\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:55:53.916541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:53.916538    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:55:54.379771 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.379739    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"]
Oct 31 03:55:54.380537 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.380520    4447 cpu_manager.go:414] "containerMap: removing container" podUID="1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce" containerName="revoke-keys"
Oct 31 03:55:54.380586 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.380538    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce" containerName="revoke-keys"
Oct 31 03:55:54.380667 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.380658    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce" containerName="revoke-keys"
Oct 31 03:55:54.384271 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.384248    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.398301 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.398278    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"]
Oct 31 03:55:54.402580 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.402564    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.424032 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.424004    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.424254 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.424037    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.424254 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.424131    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.424254 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.424232    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.424398 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.424259    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.424398 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.424291    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.424482 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.424397    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22mr4\" (UniqueName: \"kubernetes.io/projected/27ca7445-d121-4ae9-abe7-bbf2643ab453-kube-api-access-22mr4\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.424564 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.424546    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ca7445-d121-4ae9-abe7-bbf2643ab453-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.424710 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.424693    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.437158 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.437138    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1180d818-openshift-default-dockercfg-gjfz4\""
Oct 31 03:55:54.492054 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.492018    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"]
Oct 31 03:55:54.526172 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.526137    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f1c261-eea4-45ad-ad44-b0f997723e87-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.526344 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.526175    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsqjx\" (UniqueName: \"kubernetes.io/projected/e3f1c261-eea4-45ad-ad44-b0f997723e87-kube-api-access-xsqjx\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.526344 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.526315    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.526419 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.526389    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.526419 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.526409    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.526479 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.526428    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.526479 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.526449    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.526554 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.526503    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.526554 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.526550    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.526622 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.526580    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.526665 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.526640    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.526710 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.526669    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.526710 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.526694    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.526815 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.526735    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22mr4\" (UniqueName: \"kubernetes.io/projected/27ca7445-d121-4ae9-abe7-bbf2643ab453-kube-api-access-22mr4\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.526815 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.526744    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.526815 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.526791    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.527031 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.526817    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.527031 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.526849    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.527031 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.526926    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ca7445-d121-4ae9-abe7-bbf2643ab453-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.527031 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.526979    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.527031 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.527019    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.527240 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.527052    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.527519 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.527503    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ca7445-d121-4ae9-abe7-bbf2643ab453-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.529167 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.529141    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.529350 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.529329    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.541081 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.541052    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"]
Oct 31 03:55:54.569477 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.569441    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84e130-nwtzj"
Oct 31 03:55:54.569670 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.569475    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84e130-nwtzj" event={"ID":"1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce","Type":"ContainerDied","Data":"d030459e1b020977f6186fc37a7f4af8bca6045fc6a0a5a47a0a5a7d8c92d7d0"}
Oct 31 03:55:54.569670 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.569526    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d030459e1b020977f6186fc37a7f4af8bca6045fc6a0a5a47a0a5a7d8c92d7d0"
Oct 31 03:55:54.588161 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.588135    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.588294 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.588253    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22mr4\" (UniqueName: \"kubernetes.io/projected/27ca7445-d121-4ae9-abe7-bbf2643ab453-kube-api-access-22mr4\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.629430 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.629389    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsqjx\" (UniqueName: \"kubernetes.io/projected/e3f1c261-eea4-45ad-ad44-b0f997723e87-kube-api-access-xsqjx\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.629615 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.629534    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.629615 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.629567    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.629747 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.629651    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.629747 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.629689    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.629867 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.629789    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.629867 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.629822    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.630026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.629899    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.630026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.629966    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f1c261-eea4-45ad-ad44-b0f997723e87-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.630771 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.630746    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f1c261-eea4-45ad-ad44-b0f997723e87-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.632724 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.632697    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.632986 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.632966    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.633209 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.633191    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.633444 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.633419    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.639509 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.639481    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.640660 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.640621    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.692505 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.692472    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsqjx\" (UniqueName: \"kubernetes.io/projected/e3f1c261-eea4-45ad-ad44-b0f997723e87-kube-api-access-xsqjx\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.696786 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.696753    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:54.699525 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.699496    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:54.716105 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:54.716084    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:55.145036 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:55.145003    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"]
Oct 31 03:55:55.152491 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:55.151794    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"]
Oct 31 03:55:55.156566 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:55.155478    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:55:55.156566 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:55.155564    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:55:55.156566 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:55.155599    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:55:55.157248 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:55.157174    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:55:55.157342 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:55.157248    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:55:55.157342 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:55.157287    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:55:55.583822 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:55.583779    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s" event={"ID":"e3f1c261-eea4-45ad-ad44-b0f997723e87","Type":"ContainerStarted","Data":"097886db4be0667859395298df95e8ab6865a2a7aff2e8bab0da330650ae5686"}
Oct 31 03:55:55.584071 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:55.583853    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s" event={"ID":"e3f1c261-eea4-45ad-ad44-b0f997723e87","Type":"ContainerStarted","Data":"09ac262e4f52e86585d36febb749c98d6e810765a8795965c10b988d09cc7164"}
Oct 31 03:55:55.586215 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:55.586188    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj" event={"ID":"27ca7445-d121-4ae9-abe7-bbf2643ab453","Type":"ContainerStarted","Data":"9f0877ad7fe748db3532567b154c0c3b1cef30d3af87dcc3f33b034fc25175be"}
Oct 31 03:55:55.586353 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:55.586235    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj" event={"ID":"27ca7445-d121-4ae9-abe7-bbf2643ab453","Type":"ContainerStarted","Data":"5a1f551ba5e5ade7b156fa9871b3d47a66a82f42bada6d97520b0c52a43ebc66"}
Oct 31 03:55:55.700356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:55.700327    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:55:55.708092 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:55.708043    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s" podStartSLOduration=1.708016185 podStartE2EDuration="1.708016185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:55:55.705283181 +0000 UTC m=+2181.216265557" watchObservedRunningTime="2026-10-31 03:55:55.708016185 +0000 UTC m=+2181.218998561"
Oct 31 03:55:55.716602 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:55.716576    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:55:55.830503 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:55.830451    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj" podStartSLOduration=1.8304371160000001 podStartE2EDuration="1.830437116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:55:55.826082009 +0000 UTC m=+2181.337064386" watchObservedRunningTime="2026-10-31 03:55:55.830437116 +0000 UTC m=+2181.341419492"
Oct 31 03:55:56.703404 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:56.703370    4447 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 31 03:55:56.704027 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:56.703995    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj" podUID="27ca7445-d121-4ae9-abe7-bbf2643ab453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 31 03:55:56.720723 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:56.720687    4447 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 31 03:55:56.721026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:56.720995    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s" podUID="e3f1c261-eea4-45ad-ad44-b0f997723e87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 31 03:55:57.717909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:57.702279    4447 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 31 03:55:57.717909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:57.702358    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj" podUID="27ca7445-d121-4ae9-abe7-bbf2643ab453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 31 03:55:57.723618 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:57.723359    4447 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 31 03:55:57.723618 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:57.723424    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s" podUID="e3f1c261-eea4-45ad-ad44-b0f997723e87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 31 03:55:58.730909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:58.724842    4447 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 31 03:55:58.730909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:58.724925    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s" podUID="e3f1c261-eea4-45ad-ad44-b0f997723e87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 31 03:55:58.730909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:58.725381    4447 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 31 03:55:58.730909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:58.725423    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj" podUID="27ca7445-d121-4ae9-abe7-bbf2643ab453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 31 03:55:59.717910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:59.717032    4447 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 31 03:55:59.717910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:59.717099    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj" podUID="27ca7445-d121-4ae9-abe7-bbf2643ab453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 31 03:55:59.718213 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:59.718028    4447 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 31 03:55:59.718213 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:55:59.718068    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s" podUID="e3f1c261-eea4-45ad-ad44-b0f997723e87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 31 03:56:00.702910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:00.702529    4447 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 31 03:56:00.702910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:00.702589    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj" podUID="27ca7445-d121-4ae9-abe7-bbf2643ab453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 31 03:56:00.717567 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:00.717529    4447 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Oct 31 03:56:00.717857 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:00.717828    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s" podUID="e3f1c261-eea4-45ad-ad44-b0f997723e87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Oct 31 03:56:01.677827 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:01.677780    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs"]
Oct 31 03:56:01.802508 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:01.802471    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599"]
Oct 31 03:56:01.881514 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:01.881483    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:56:01.882230 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:01.881924    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:56:01.883468 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:01.883446    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:56:01.885781 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:01.885761    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:56:01.886157 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:01.886139    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:56:01.886837 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:01.886817    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:56:06.678750 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:06.678710    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs" podUID="c75320fc-6fe9-40e0-a67a-f70757ffd71a" containerName="payload-processing" containerID="cri-o://09c8f3468aef28571e01f18fd10acbbfc369bce1a6578ced807d56c321e18960" gracePeriod=25
Oct 31 03:56:06.804491 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:06.804449    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599" podUID="cd0b36fc-225c-4452-8d77-0b28c1679c7a" containerName="payload-pre-processing" containerID="cri-o://bf693d926f1d1cd6d2069cc0cfb09a5e184782f7ee0cae7f59475d9b5843df7e" gracePeriod=25
Oct 31 03:56:06.970363 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:06.970331    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs"
Oct 31 03:56:07.073362 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.073333    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75320fc-6fe9-40e0-a67a-f70757ffd71a-kube-api-access-qqpnl\" (UniqueName: \"kubernetes.io/projected/c75320fc-6fe9-40e0-a67a-f70757ffd71a-kube-api-access-qqpnl\") pod \"c75320fc-6fe9-40e0-a67a-f70757ffd71a\" (UID: \"c75320fc-6fe9-40e0-a67a-f70757ffd71a\") "
Oct 31 03:56:07.073597 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.073568    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c75320fc-6fe9-40e0-a67a-f70757ffd71a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75320fc-6fe9-40e0-a67a-f70757ffd71a-plugins-config-volume\") pod \"c75320fc-6fe9-40e0-a67a-f70757ffd71a\" (UID: \"c75320fc-6fe9-40e0-a67a-f70757ffd71a\") "
Oct 31 03:56:07.073956 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.073929    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c75320fc-6fe9-40e0-a67a-f70757ffd71a-plugins-config-volume" pod "c75320fc-6fe9-40e0-a67a-f70757ffd71a" (UID: "c75320fc-6fe9-40e0-a67a-f70757ffd71a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:07.074150 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.074134    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75320fc-6fe9-40e0-a67a-f70757ffd71a-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:56:07.075493 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.075471    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75320fc-6fe9-40e0-a67a-f70757ffd71a-kube-api-access-qqpnl" pod "c75320fc-6fe9-40e0-a67a-f70757ffd71a" (UID: "c75320fc-6fe9-40e0-a67a-f70757ffd71a"). InnerVolumeSpecName "kube-api-access-qqpnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:07.088063 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.088039    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599"
Oct 31 03:56:07.174948 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.174919    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd0b36fc-225c-4452-8d77-0b28c1679c7a-kube-api-access-z4q99\" (UniqueName: \"kubernetes.io/projected/cd0b36fc-225c-4452-8d77-0b28c1679c7a-kube-api-access-z4q99\") pod \"cd0b36fc-225c-4452-8d77-0b28c1679c7a\" (UID: \"cd0b36fc-225c-4452-8d77-0b28c1679c7a\") "
Oct 31 03:56:07.175109 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.175098    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd0b36fc-225c-4452-8d77-0b28c1679c7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0b36fc-225c-4452-8d77-0b28c1679c7a-plugins-config-volume\") pod \"cd0b36fc-225c-4452-8d77-0b28c1679c7a\" (UID: \"cd0b36fc-225c-4452-8d77-0b28c1679c7a\") "
Oct 31 03:56:07.175435 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.175396    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd0b36fc-225c-4452-8d77-0b28c1679c7a-plugins-config-volume" pod "cd0b36fc-225c-4452-8d77-0b28c1679c7a" (UID: "cd0b36fc-225c-4452-8d77-0b28c1679c7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:07.175746 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.175556    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0b36fc-225c-4452-8d77-0b28c1679c7a-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:56:07.175746 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.175569    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqpnl\" (UniqueName: \"kubernetes.io/projected/c75320fc-6fe9-40e0-a67a-f70757ffd71a-kube-api-access-qqpnl\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:56:07.177224 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.177203    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0b36fc-225c-4452-8d77-0b28c1679c7a-kube-api-access-z4q99" pod "cd0b36fc-225c-4452-8d77-0b28c1679c7a" (UID: "cd0b36fc-225c-4452-8d77-0b28c1679c7a"). InnerVolumeSpecName "kube-api-access-z4q99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:07.276799 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.276765    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4q99\" (UniqueName: \"kubernetes.io/projected/cd0b36fc-225c-4452-8d77-0b28c1679c7a-kube-api-access-z4q99\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:56:07.742582 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.742548    4447 generic.go:356] "Generic (PLEG): container finished" podID="cd0b36fc-225c-4452-8d77-0b28c1679c7a" containerID="bf693d926f1d1cd6d2069cc0cfb09a5e184782f7ee0cae7f59475d9b5843df7e" exitCode=0
Oct 31 03:56:07.743067 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.742599    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599"
Oct 31 03:56:07.743067 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.742637    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599" event={"ID":"cd0b36fc-225c-4452-8d77-0b28c1679c7a","Type":"ContainerDied","Data":"bf693d926f1d1cd6d2069cc0cfb09a5e184782f7ee0cae7f59475d9b5843df7e"}
Oct 31 03:56:07.743067 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.742688    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599" event={"ID":"cd0b36fc-225c-4452-8d77-0b28c1679c7a","Type":"ContainerDied","Data":"435b3e1ea9246cc947d11de20b5e0a6da8b9301d6f5e7372c556bcc6392b55e6"}
Oct 31 03:56:07.743067 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.742716    4447 scope.go:122] "RemoveContainer" containerID="bf693d926f1d1cd6d2069cc0cfb09a5e184782f7ee0cae7f59475d9b5843df7e"
Oct 31 03:56:07.744199 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.744176    4447 generic.go:356] "Generic (PLEG): container finished" podID="c75320fc-6fe9-40e0-a67a-f70757ffd71a" containerID="09c8f3468aef28571e01f18fd10acbbfc369bce1a6578ced807d56c321e18960" exitCode=0
Oct 31 03:56:07.744490 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.744211    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs" event={"ID":"c75320fc-6fe9-40e0-a67a-f70757ffd71a","Type":"ContainerDied","Data":"09c8f3468aef28571e01f18fd10acbbfc369bce1a6578ced807d56c321e18960"}
Oct 31 03:56:07.744490 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.744270    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs" event={"ID":"c75320fc-6fe9-40e0-a67a-f70757ffd71a","Type":"ContainerDied","Data":"390562a946cec6c6f4544b52b43c147b906bf836e6942eb432383031db293d52"}
Oct 31 03:56:07.744490 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.744220    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs"
Oct 31 03:56:07.754406 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.754385    4447 scope.go:122] "RemoveContainer" containerID="bf693d926f1d1cd6d2069cc0cfb09a5e184782f7ee0cae7f59475d9b5843df7e"
Oct 31 03:56:07.754761 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:56:07.754738    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf693d926f1d1cd6d2069cc0cfb09a5e184782f7ee0cae7f59475d9b5843df7e\": container with ID starting with bf693d926f1d1cd6d2069cc0cfb09a5e184782f7ee0cae7f59475d9b5843df7e not found: ID does not exist" containerID="bf693d926f1d1cd6d2069cc0cfb09a5e184782f7ee0cae7f59475d9b5843df7e"
Oct 31 03:56:07.754843 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.754774    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf693d926f1d1cd6d2069cc0cfb09a5e184782f7ee0cae7f59475d9b5843df7e"} err="failed to get container status \"bf693d926f1d1cd6d2069cc0cfb09a5e184782f7ee0cae7f59475d9b5843df7e\": rpc error: code = NotFound desc = could not find container \"bf693d926f1d1cd6d2069cc0cfb09a5e184782f7ee0cae7f59475d9b5843df7e\": container with ID starting with bf693d926f1d1cd6d2069cc0cfb09a5e184782f7ee0cae7f59475d9b5843df7e not found: ID does not exist"
Oct 31 03:56:07.754843 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.754798    4447 scope.go:122] "RemoveContainer" containerID="09c8f3468aef28571e01f18fd10acbbfc369bce1a6578ced807d56c321e18960"
Oct 31 03:56:07.765812 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.765791    4447 scope.go:122] "RemoveContainer" containerID="09c8f3468aef28571e01f18fd10acbbfc369bce1a6578ced807d56c321e18960"
Oct 31 03:56:07.766179 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:56:07.766160    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09c8f3468aef28571e01f18fd10acbbfc369bce1a6578ced807d56c321e18960\": container with ID starting with 09c8f3468aef28571e01f18fd10acbbfc369bce1a6578ced807d56c321e18960 not found: ID does not exist" containerID="09c8f3468aef28571e01f18fd10acbbfc369bce1a6578ced807d56c321e18960"
Oct 31 03:56:07.766248 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.766189    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c8f3468aef28571e01f18fd10acbbfc369bce1a6578ced807d56c321e18960"} err="failed to get container status \"09c8f3468aef28571e01f18fd10acbbfc369bce1a6578ced807d56c321e18960\": rpc error: code = NotFound desc = could not find container \"09c8f3468aef28571e01f18fd10acbbfc369bce1a6578ced807d56c321e18960\": container with ID starting with 09c8f3468aef28571e01f18fd10acbbfc369bce1a6578ced807d56c321e18960 not found: ID does not exist"
Oct 31 03:56:07.911026 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.910990    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599"]
Oct 31 03:56:07.960434 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:07.960401    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-84e130-5d9fc469cf-dz599"]
Oct 31 03:56:08.024803 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.024765    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs"]
Oct 31 03:56:08.029041 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.029014    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-84e130-7588fc9bb9-dgkxs"]
Oct 31 03:56:08.071873 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.071835    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt"]
Oct 31 03:56:08.072972 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.072955    4447 cpu_manager.go:414] "containerMap: removing container" podUID="cd0b36fc-225c-4452-8d77-0b28c1679c7a" containerName="payload-pre-processing"
Oct 31 03:56:08.073024 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.072975    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0b36fc-225c-4452-8d77-0b28c1679c7a" containerName="payload-pre-processing"
Oct 31 03:56:08.073024 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.072992    4447 cpu_manager.go:414] "containerMap: removing container" podUID="c75320fc-6fe9-40e0-a67a-f70757ffd71a" containerName="payload-processing"
Oct 31 03:56:08.073024 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.073000    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75320fc-6fe9-40e0-a67a-f70757ffd71a" containerName="payload-processing"
Oct 31 03:56:08.073215 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.073204    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0b36fc-225c-4452-8d77-0b28c1679c7a" containerName="payload-pre-processing"
Oct 31 03:56:08.073248 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.073216    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75320fc-6fe9-40e0-a67a-f70757ffd71a" containerName="payload-processing"
Oct 31 03:56:08.078447 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.078427    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt"
Oct 31 03:56:08.131700 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.131673    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:56:08.131869 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.131745    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 03:56:08.133336 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.133317    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt"]
Oct 31 03:56:08.190685 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.190642    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d924e9-f7d4-435f-b07b-68263ee2ab69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt\" (UID: \"c6d924e9-f7d4-435f-b07b-68263ee2ab69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt"
Oct 31 03:56:08.190866 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.190832    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d924e9-f7d4-435f-b07b-68263ee2ab69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt\" (UID: \"c6d924e9-f7d4-435f-b07b-68263ee2ab69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt"
Oct 31 03:56:08.190942 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.190865    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22k6d\" (UniqueName: \"kubernetes.io/projected/c6d924e9-f7d4-435f-b07b-68263ee2ab69-kube-api-access-22k6d\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt\" (UID: \"c6d924e9-f7d4-435f-b07b-68263ee2ab69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt"
Oct 31 03:56:08.292383 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.292291    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d924e9-f7d4-435f-b07b-68263ee2ab69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt\" (UID: \"c6d924e9-f7d4-435f-b07b-68263ee2ab69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt"
Oct 31 03:56:08.292541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.292441    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d924e9-f7d4-435f-b07b-68263ee2ab69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt\" (UID: \"c6d924e9-f7d4-435f-b07b-68263ee2ab69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt"
Oct 31 03:56:08.292541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.292470    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22k6d\" (UniqueName: \"kubernetes.io/projected/c6d924e9-f7d4-435f-b07b-68263ee2ab69-kube-api-access-22k6d\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt\" (UID: \"c6d924e9-f7d4-435f-b07b-68263ee2ab69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt"
Oct 31 03:56:08.293324 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.293226    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d924e9-f7d4-435f-b07b-68263ee2ab69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt\" (UID: \"c6d924e9-f7d4-435f-b07b-68263ee2ab69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt"
Oct 31 03:56:08.295048 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.295025    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d924e9-f7d4-435f-b07b-68263ee2ab69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt\" (UID: \"c6d924e9-f7d4-435f-b07b-68263ee2ab69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt"
Oct 31 03:56:08.353657 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.353626    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22k6d\" (UniqueName: \"kubernetes.io/projected/c6d924e9-f7d4-435f-b07b-68263ee2ab69-kube-api-access-22k6d\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt\" (UID: \"c6d924e9-f7d4-435f-b07b-68263ee2ab69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt"
Oct 31 03:56:08.392898 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.392849    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt"
Oct 31 03:56:08.611217 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.611134    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c75320fc-6fe9-40e0-a67a-f70757ffd71a" path="/var/lib/kubelet/pods/c75320fc-6fe9-40e0-a67a-f70757ffd71a/volumes"
Oct 31 03:56:08.611679 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.611664    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd0b36fc-225c-4452-8d77-0b28c1679c7a" path="/var/lib/kubelet/pods/cd0b36fc-225c-4452-8d77-0b28c1679c7a/volumes"
Oct 31 03:56:08.778371 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:08.778341    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt"]
Oct 31 03:56:08.778767 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:56:08.778698    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d924e9_f7d4_435f_b07b_68263ee2ab69.slice/crio-cfde3568cfa22795b60bb56282f4d064a0c08f76de823deb7ad172613a54c5d0 WatchSource:0}: Error finding container cfde3568cfa22795b60bb56282f4d064a0c08f76de823deb7ad172613a54c5d0: Status 404 returned error can't find the container with id cfde3568cfa22795b60bb56282f4d064a0c08f76de823deb7ad172613a54c5d0
Oct 31 03:56:09.522973 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.522942    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"]
Oct 31 03:56:09.527599 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.527579    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.610998 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.610956    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.610998 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.610989    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.611204 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.611090    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c96k8\" (UniqueName: \"kubernetes.io/projected/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-kube-api-access-c96k8\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.611204 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.611128    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.611204 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.611182    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.611343 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.611256    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.611343 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.611291    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.611410 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.611391    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.611547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.611525    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.713329 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.713297    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.713518 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.713354    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.713518 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.713373    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.713518 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.713405    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c96k8\" (UniqueName: \"kubernetes.io/projected/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-kube-api-access-c96k8\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.713518 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.713426    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.713518 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.713497    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.713794 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.713563    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.713794 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.713599    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.713794 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.713692    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.713794 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.713736    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.713794 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.713776    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.714119 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.713806    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.714119 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.714085    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.714235 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.714212    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.716021 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.716003    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.716355 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.716335    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.746291 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.746261    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"]
Oct 31 03:56:09.768170 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.768145    4447 generic.go:356] "Generic (PLEG): container finished" podID="c6d924e9-f7d4-435f-b07b-68263ee2ab69" containerID="32e5392d04364ca73f6ac9a3da61cdc4033851d1218bd7d861077f545fbdfcd1" exitCode=0
Oct 31 03:56:09.768305 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.768229    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt" event={"ID":"c6d924e9-f7d4-435f-b07b-68263ee2ab69","Type":"ContainerDied","Data":"32e5392d04364ca73f6ac9a3da61cdc4033851d1218bd7d861077f545fbdfcd1"}
Oct 31 03:56:09.768305 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.768294    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt" event={"ID":"c6d924e9-f7d4-435f-b07b-68263ee2ab69","Type":"ContainerStarted","Data":"cfde3568cfa22795b60bb56282f4d064a0c08f76de823deb7ad172613a54c5d0"}
Oct 31 03:56:09.775525 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.775470    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.775624 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.775545    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c96k8\" (UniqueName: \"kubernetes.io/projected/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-kube-api-access-c96k8\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:09.878578 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:09.878549    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:10.362443 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:56:10.362402    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda02c6569_f6f4_41ac_bcb1_cd42873c13b2.slice/crio-0fa2a18e2d0f294c5dfb22659e1105e1cf3ff1028afc3f7d1b106c9f5e35f5dc WatchSource:0}: Error finding container 0fa2a18e2d0f294c5dfb22659e1105e1cf3ff1028afc3f7d1b106c9f5e35f5dc: Status 404 returned error can't find the container with id 0fa2a18e2d0f294c5dfb22659e1105e1cf3ff1028afc3f7d1b106c9f5e35f5dc
Oct 31 03:56:10.363074 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:10.363047    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"]
Oct 31 03:56:10.365630 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:10.365597    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:56:10.365705 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:10.365663    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:56:10.365705 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:10.365691    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:56:10.777953 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:10.777921    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh" event={"ID":"a02c6569-f6f4-41ac-bcb1-cd42873c13b2","Type":"ContainerStarted","Data":"7e483ad7884cbc2c841b46bb02a0dfbde8f6200d94708c0c38282f616d74d5b9"}
Oct 31 03:56:10.778158 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:10.777977    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh" event={"ID":"a02c6569-f6f4-41ac-bcb1-cd42873c13b2","Type":"ContainerStarted","Data":"0fa2a18e2d0f294c5dfb22659e1105e1cf3ff1028afc3f7d1b106c9f5e35f5dc"}
Oct 31 03:56:10.885927 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:10.881995    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:10.901347 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:10.901287    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh" podStartSLOduration=1.901268781 podStartE2EDuration="1.901268781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:10.897307242 +0000 UTC m=+2196.408289620" watchObservedRunningTime="2026-10-31 03:56:10.901268781 +0000 UTC m=+2196.412251161"
Oct 31 03:56:10.941237 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:10.941212    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt"
Oct 31 03:56:11.032293 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:11.032207    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6d924e9-f7d4-435f-b07b-68263ee2ab69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d924e9-f7d4-435f-b07b-68263ee2ab69-maas-api-xks-ca\") pod \"c6d924e9-f7d4-435f-b07b-68263ee2ab69\" (UID: \"c6d924e9-f7d4-435f-b07b-68263ee2ab69\") "
Oct 31 03:56:11.032449 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:11.032415    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d924e9-f7d4-435f-b07b-68263ee2ab69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d924e9-f7d4-435f-b07b-68263ee2ab69-maas-api-openshift-ca\") pod \"c6d924e9-f7d4-435f-b07b-68263ee2ab69\" (UID: \"c6d924e9-f7d4-435f-b07b-68263ee2ab69\") "
Oct 31 03:56:11.032449 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:11.032438    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d924e9-f7d4-435f-b07b-68263ee2ab69-kube-api-access-22k6d\" (UniqueName: \"kubernetes.io/projected/c6d924e9-f7d4-435f-b07b-68263ee2ab69-kube-api-access-22k6d\") pod \"c6d924e9-f7d4-435f-b07b-68263ee2ab69\" (UID: \"c6d924e9-f7d4-435f-b07b-68263ee2ab69\") "
Oct 31 03:56:11.032793 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:11.032761    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d924e9-f7d4-435f-b07b-68263ee2ab69-maas-api-openshift-ca" pod "c6d924e9-f7d4-435f-b07b-68263ee2ab69" (UID: "c6d924e9-f7d4-435f-b07b-68263ee2ab69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:11.033117 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:11.033099    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d924e9-f7d4-435f-b07b-68263ee2ab69-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:56:11.034599 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:11.034576    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6d924e9-f7d4-435f-b07b-68263ee2ab69-maas-api-xks-ca" pod "c6d924e9-f7d4-435f-b07b-68263ee2ab69" (UID: "c6d924e9-f7d4-435f-b07b-68263ee2ab69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:56:11.034692 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:11.034620    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d924e9-f7d4-435f-b07b-68263ee2ab69-kube-api-access-22k6d" pod "c6d924e9-f7d4-435f-b07b-68263ee2ab69" (UID: "c6d924e9-f7d4-435f-b07b-68263ee2ab69"). InnerVolumeSpecName "kube-api-access-22k6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:11.134701 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:11.134667    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22k6d\" (UniqueName: \"kubernetes.io/projected/c6d924e9-f7d4-435f-b07b-68263ee2ab69-kube-api-access-22k6d\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:56:11.134701 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:11.134696    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d924e9-f7d4-435f-b07b-68263ee2ab69-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:56:11.809088 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:11.809049    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt" event={"ID":"c6d924e9-f7d4-435f-b07b-68263ee2ab69","Type":"ContainerDied","Data":"cfde3568cfa22795b60bb56282f4d064a0c08f76de823deb7ad172613a54c5d0"}
Oct 31 03:56:11.809317 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:11.809156    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfde3568cfa22795b60bb56282f4d064a0c08f76de823deb7ad172613a54c5d0"
Oct 31 03:56:11.809317 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:11.809058    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt"
Oct 31 03:56:11.902504 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:11.902468    4447 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 31 03:56:11.903049 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:11.902540    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh" podUID="a02c6569-f6f4-41ac-bcb1-cd42873c13b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 31 03:56:12.893910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:12.880310    4447 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 31 03:56:12.893910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:12.880377    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh" podUID="a02c6569-f6f4-41ac-bcb1-cd42873c13b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 31 03:56:13.894303 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:13.894265    4447 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 31 03:56:13.894911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:13.894857    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh" podUID="a02c6569-f6f4-41ac-bcb1-cd42873c13b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 31 03:56:14.898911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:14.898179    4447 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 31 03:56:14.898911 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:14.898239    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh" podUID="a02c6569-f6f4-41ac-bcb1-cd42873c13b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 31 03:56:15.893373 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:15.893328    4447 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 31 03:56:15.893646 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:15.893614    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh" podUID="a02c6569-f6f4-41ac-bcb1-cd42873c13b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 31 03:56:16.884321 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:16.884225    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:16.886502 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:16.884730    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:16.886502 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:16.885561    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:56:19.004209 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:56:19.004169    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:56:24.654101 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.654062    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"]
Oct 31 03:56:24.654804 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.654790    4447 cpu_manager.go:414] "containerMap: removing container" podUID="c6d924e9-f7d4-435f-b07b-68263ee2ab69" containerName="revoke-keys"
Oct 31 03:56:24.654848 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.654806    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d924e9-f7d4-435f-b07b-68263ee2ab69" containerName="revoke-keys"
Oct 31 03:56:24.654987 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.654976    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d924e9-f7d4-435f-b07b-68263ee2ab69" containerName="revoke-keys"
Oct 31 03:56:24.658921 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.658903    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.767004 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.766970    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"]
Oct 31 03:56:24.827450 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.827410    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.827450 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.827452    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k78g\" (UniqueName: \"kubernetes.io/projected/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-kube-api-access-6k78g\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.827697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.827559    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.827697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.827620    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.827697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.827641    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.827807 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.827714    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.827807 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.827764    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.827978 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.827959    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.828055 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.828001    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.929683 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.929603    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k78g\" (UniqueName: \"kubernetes.io/projected/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-kube-api-access-6k78g\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.929683 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.929663    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.929985 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.929709    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.929985 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.929731    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.929985 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.929780    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.929985 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.929833    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.929985 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.929961    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.930254 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.929994    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.930254 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.930113    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.930349 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.930288    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-data\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.930401 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.930355    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-credential-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.930401 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.930385    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-workload-certs\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.930669 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.930483    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-workload-socket\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.930669 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.930651    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istiod-ca-cert\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.932228 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.932206    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-envoy\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.932765 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.932748    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-podinfo\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.991426 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.991390    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-token\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:24.991573 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:24.991515    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k78g\" (UniqueName: \"kubernetes.io/projected/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-kube-api-access-6k78g\") pod \"e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") " pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:25.273868 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:25.273830    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:25.707964 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:25.707934    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"]
Oct 31 03:56:25.741845 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:56:25.741819    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcada5ab7_e6b6_45e8_85ce_53c9c4d6c10f.slice/crio-72a8947be904e39f009aacc069ebb6c6554bd3f1cdbdc4eb283bf20deec49b78 WatchSource:0}: Error finding container 72a8947be904e39f009aacc069ebb6c6554bd3f1cdbdc4eb283bf20deec49b78: Status 404 returned error can't find the container with id 72a8947be904e39f009aacc069ebb6c6554bd3f1cdbdc4eb283bf20deec49b78
Oct 31 03:56:25.744548 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:25.744513    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:56:25.744657 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:25.744598    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:56:25.744657 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:25.744652    4447 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 03:56:25.936366 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:25.936333    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m" event={"ID":"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f","Type":"ContainerStarted","Data":"b250474413cfbbf3a78cd51a679c7816522ac53451fd6624685caba20cac8f6e"}
Oct 31 03:56:25.936540 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:25.936426    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m" event={"ID":"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f","Type":"ContainerStarted","Data":"72a8947be904e39f009aacc069ebb6c6554bd3f1cdbdc4eb283bf20deec49b78"}
Oct 31 03:56:26.058936 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:26.058809    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m" podStartSLOduration=2.058795447 podStartE2EDuration="2.058795447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:26.056626842 +0000 UTC m=+2211.567609219" watchObservedRunningTime="2026-10-31 03:56:26.058795447 +0000 UTC m=+2211.569777824"
Oct 31 03:56:26.274807 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:26.274778    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:26.276334 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:26.276305    4447 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 31 03:56:26.276449 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:26.276351    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m" podUID="cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 31 03:56:27.298532 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:27.298494    4447 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 31 03:56:27.299159 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:27.299123    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m" podUID="cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 31 03:56:28.299068 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:28.299025    4447 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 31 03:56:28.299558 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:28.299108    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m" podUID="cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 31 03:56:29.298923 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:29.293544    4447 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 31 03:56:29.298923 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:29.293601    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m" podUID="cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 31 03:56:30.291910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:30.288011    4447 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 31 03:56:30.291910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:30.288113    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m" podUID="cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 31 03:56:31.294910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:31.294314    4447 patch_prober.go:29] interesting pod/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 31 03:56:31.294910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:31.294381    4447 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m" podUID="cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 31 03:56:32.278717 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:32.278687    4447 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:32.279202 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:32.279183    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:32.279861 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:32.279841    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:56:40.796788 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:40.796745    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf"]
Oct 31 03:56:40.847147 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:40.847114    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc91a3-pl9kf"]
Oct 31 03:56:42.610972 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:42.610937    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aab3ee5-2dbc-4227-9267-4f9c05857305" path="/var/lib/kubelet/pods/4aab3ee5-2dbc-4227-9267-4f9c05857305/volumes"
Oct 31 03:56:49.003485 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:56:49.003448    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:56:49.024398 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:49.024356    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp"]
Oct 31 03:56:49.030547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:49.030530    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp"
Oct 31 03:56:49.090035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:49.089985    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp"]
Oct 31 03:56:49.132341 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:49.132309    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:56:49.132491 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:49.132475    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 03:56:49.142608 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:49.142535    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8ae678-217a-4501-8cd1-646f7a169f3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp\" (UID: \"3c8ae678-217a-4501-8cd1-646f7a169f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp"
Oct 31 03:56:49.142785 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:49.142650    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8ae678-217a-4501-8cd1-646f7a169f3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp\" (UID: \"3c8ae678-217a-4501-8cd1-646f7a169f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp"
Oct 31 03:56:49.142838 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:49.142790    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz9d8\" (UniqueName: \"kubernetes.io/projected/3c8ae678-217a-4501-8cd1-646f7a169f3e-kube-api-access-bz9d8\") pod \"maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp\" (UID: \"3c8ae678-217a-4501-8cd1-646f7a169f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp"
Oct 31 03:56:49.244536 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:49.244501    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz9d8\" (UniqueName: \"kubernetes.io/projected/3c8ae678-217a-4501-8cd1-646f7a169f3e-kube-api-access-bz9d8\") pod \"maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp\" (UID: \"3c8ae678-217a-4501-8cd1-646f7a169f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp"
Oct 31 03:56:49.244711 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:49.244668    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8ae678-217a-4501-8cd1-646f7a169f3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp\" (UID: \"3c8ae678-217a-4501-8cd1-646f7a169f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp"
Oct 31 03:56:49.244757 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:49.244710    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8ae678-217a-4501-8cd1-646f7a169f3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp\" (UID: \"3c8ae678-217a-4501-8cd1-646f7a169f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp"
Oct 31 03:56:49.245417 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:49.245392    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8ae678-217a-4501-8cd1-646f7a169f3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp\" (UID: \"3c8ae678-217a-4501-8cd1-646f7a169f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp"
Oct 31 03:56:49.247315 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:49.247283    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8ae678-217a-4501-8cd1-646f7a169f3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp\" (UID: \"3c8ae678-217a-4501-8cd1-646f7a169f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp"
Oct 31 03:56:49.306144 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:49.306074    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz9d8\" (UniqueName: \"kubernetes.io/projected/3c8ae678-217a-4501-8cd1-646f7a169f3e-kube-api-access-bz9d8\") pod \"maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp\" (UID: \"3c8ae678-217a-4501-8cd1-646f7a169f3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp"
Oct 31 03:56:49.344601 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:49.344570    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp"
Oct 31 03:56:49.729456 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:56:49.729422    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8ae678_217a_4501_8cd1_646f7a169f3e.slice/crio-c74dcd9e582ed4a4d11183be1b5e09172c99a020e71352501f80b29bf6032049 WatchSource:0}: Error finding container c74dcd9e582ed4a4d11183be1b5e09172c99a020e71352501f80b29bf6032049: Status 404 returned error can't find the container with id c74dcd9e582ed4a4d11183be1b5e09172c99a020e71352501f80b29bf6032049
Oct 31 03:56:49.731054 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:49.731030    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp"]
Oct 31 03:56:50.049143 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:50.049105    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-h526p"]
Oct 31 03:56:50.053877 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:50.053859    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-h526p"
Oct 31 03:56:50.113412 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:50.113377    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-h526p"]
Oct 31 03:56:50.159115 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:50.159084    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edacc995-47b9-4ed7-9a68-7aa8fdce1ce0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1180d818-h526p\" (UID: \"edacc995-47b9-4ed7-9a68-7aa8fdce1ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-h526p"
Oct 31 03:56:50.159311 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:50.159158    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edacc995-47b9-4ed7-9a68-7aa8fdce1ce0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1180d818-h526p\" (UID: \"edacc995-47b9-4ed7-9a68-7aa8fdce1ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-h526p"
Oct 31 03:56:50.159311 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:50.159296    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl2lj\" (UniqueName: \"kubernetes.io/projected/edacc995-47b9-4ed7-9a68-7aa8fdce1ce0-kube-api-access-xl2lj\") pod \"maas-api-revoke-keys-e2e-mt-1180d818-h526p\" (UID: \"edacc995-47b9-4ed7-9a68-7aa8fdce1ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-h526p"
Oct 31 03:56:50.198983 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:50.198940    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp" event={"ID":"3c8ae678-217a-4501-8cd1-646f7a169f3e","Type":"ContainerStarted","Data":"b02d5b0e7885f13b3138723ed78fb071f6486b537d6795aadd6259af6e48b760"}
Oct 31 03:56:50.199170 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:50.199001    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp" event={"ID":"3c8ae678-217a-4501-8cd1-646f7a169f3e","Type":"ContainerStarted","Data":"c74dcd9e582ed4a4d11183be1b5e09172c99a020e71352501f80b29bf6032049"}
Oct 31 03:56:50.261276 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:50.261180    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edacc995-47b9-4ed7-9a68-7aa8fdce1ce0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1180d818-h526p\" (UID: \"edacc995-47b9-4ed7-9a68-7aa8fdce1ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-h526p"
Oct 31 03:56:50.261440 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:50.261292    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edacc995-47b9-4ed7-9a68-7aa8fdce1ce0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1180d818-h526p\" (UID: \"edacc995-47b9-4ed7-9a68-7aa8fdce1ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-h526p"
Oct 31 03:56:50.261440 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:50.261399    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl2lj\" (UniqueName: \"kubernetes.io/projected/edacc995-47b9-4ed7-9a68-7aa8fdce1ce0-kube-api-access-xl2lj\") pod \"maas-api-revoke-keys-e2e-mt-1180d818-h526p\" (UID: \"edacc995-47b9-4ed7-9a68-7aa8fdce1ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-h526p"
Oct 31 03:56:50.262031 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:50.262009    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edacc995-47b9-4ed7-9a68-7aa8fdce1ce0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1180d818-h526p\" (UID: \"edacc995-47b9-4ed7-9a68-7aa8fdce1ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-h526p"
Oct 31 03:56:50.263675 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:50.263656    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edacc995-47b9-4ed7-9a68-7aa8fdce1ce0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1180d818-h526p\" (UID: \"edacc995-47b9-4ed7-9a68-7aa8fdce1ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-h526p"
Oct 31 03:56:50.317344 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:50.317289    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp" podStartSLOduration=2.317274958 podStartE2EDuration="2.317274958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:50.315279892 +0000 UTC m=+2235.826262270" watchObservedRunningTime="2026-10-31 03:56:50.317274958 +0000 UTC m=+2235.828257335"
Oct 31 03:56:50.322875 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:50.322849    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl2lj\" (UniqueName: \"kubernetes.io/projected/edacc995-47b9-4ed7-9a68-7aa8fdce1ce0-kube-api-access-xl2lj\") pod \"maas-api-revoke-keys-e2e-mt-1180d818-h526p\" (UID: \"edacc995-47b9-4ed7-9a68-7aa8fdce1ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-h526p"
Oct 31 03:56:50.369232 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:50.369205    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-h526p"
Oct 31 03:56:50.749926 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:56:50.749873    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedacc995_47b9_4ed7_9a68_7aa8fdce1ce0.slice/crio-4a3711c7fd0ec35479e0d4462b4c196b4abed24450ecf795c6bc6cbb7e6937a9 WatchSource:0}: Error finding container 4a3711c7fd0ec35479e0d4462b4c196b4abed24450ecf795c6bc6cbb7e6937a9: Status 404 returned error can't find the container with id 4a3711c7fd0ec35479e0d4462b4c196b4abed24450ecf795c6bc6cbb7e6937a9
Oct 31 03:56:50.750296 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:50.750270    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-h526p"]
Oct 31 03:56:51.208472 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:51.208442    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-h526p" event={"ID":"edacc995-47b9-4ed7-9a68-7aa8fdce1ce0","Type":"ContainerStarted","Data":"53f147f5034354c7d8667a89d04af3cc012fe3f83ed3af22683e86b6ee2a7ba2"}
Oct 31 03:56:51.208866 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:51.208500    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-h526p" event={"ID":"edacc995-47b9-4ed7-9a68-7aa8fdce1ce0","Type":"ContainerStarted","Data":"4a3711c7fd0ec35479e0d4462b4c196b4abed24450ecf795c6bc6cbb7e6937a9"}
Oct 31 03:56:51.210136 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:51.210114    4447 generic.go:356] "Generic (PLEG): container finished" podID="3c8ae678-217a-4501-8cd1-646f7a169f3e" containerID="b02d5b0e7885f13b3138723ed78fb071f6486b537d6795aadd6259af6e48b760" exitCode=0
Oct 31 03:56:51.210233 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:51.210163    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp" event={"ID":"3c8ae678-217a-4501-8cd1-646f7a169f3e","Type":"ContainerDied","Data":"b02d5b0e7885f13b3138723ed78fb071f6486b537d6795aadd6259af6e48b760"}
Oct 31 03:56:51.331362 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:51.331312    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-h526p" podStartSLOduration=1.3312980589999999 podStartE2EDuration="1.331298059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:51.327477505 +0000 UTC m=+2236.838459883" watchObservedRunningTime="2026-10-31 03:56:51.331298059 +0000 UTC m=+2236.842280434"
Oct 31 03:56:52.228243 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:52.228201    4447 generic.go:356] "Generic (PLEG): container finished" podID="edacc995-47b9-4ed7-9a68-7aa8fdce1ce0" containerID="53f147f5034354c7d8667a89d04af3cc012fe3f83ed3af22683e86b6ee2a7ba2" exitCode=0
Oct 31 03:56:52.229387 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:52.229349    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-h526p" event={"ID":"edacc995-47b9-4ed7-9a68-7aa8fdce1ce0","Type":"ContainerDied","Data":"53f147f5034354c7d8667a89d04af3cc012fe3f83ed3af22683e86b6ee2a7ba2"}
Oct 31 03:56:52.391739 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:52.391712    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp"
Oct 31 03:56:52.409155 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:52.409129    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c8ae678-217a-4501-8cd1-646f7a169f3e-kube-api-access-bz9d8\" (UniqueName: \"kubernetes.io/projected/3c8ae678-217a-4501-8cd1-646f7a169f3e-kube-api-access-bz9d8\") pod \"3c8ae678-217a-4501-8cd1-646f7a169f3e\" (UID: \"3c8ae678-217a-4501-8cd1-646f7a169f3e\") "
Oct 31 03:56:52.409317 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:52.409217    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c8ae678-217a-4501-8cd1-646f7a169f3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8ae678-217a-4501-8cd1-646f7a169f3e-maas-api-xks-ca\") pod \"3c8ae678-217a-4501-8cd1-646f7a169f3e\" (UID: \"3c8ae678-217a-4501-8cd1-646f7a169f3e\") "
Oct 31 03:56:52.409317 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:52.409307    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c8ae678-217a-4501-8cd1-646f7a169f3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8ae678-217a-4501-8cd1-646f7a169f3e-maas-api-openshift-ca\") pod \"3c8ae678-217a-4501-8cd1-646f7a169f3e\" (UID: \"3c8ae678-217a-4501-8cd1-646f7a169f3e\") "
Oct 31 03:56:52.409692 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:52.409663    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c8ae678-217a-4501-8cd1-646f7a169f3e-maas-api-openshift-ca" pod "3c8ae678-217a-4501-8cd1-646f7a169f3e" (UID: "3c8ae678-217a-4501-8cd1-646f7a169f3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:52.409865 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:52.409847    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8ae678-217a-4501-8cd1-646f7a169f3e-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:56:52.412022 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:52.411991    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c8ae678-217a-4501-8cd1-646f7a169f3e-maas-api-xks-ca" pod "3c8ae678-217a-4501-8cd1-646f7a169f3e" (UID: "3c8ae678-217a-4501-8cd1-646f7a169f3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:56:52.412114 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:52.411997    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8ae678-217a-4501-8cd1-646f7a169f3e-kube-api-access-bz9d8" pod "3c8ae678-217a-4501-8cd1-646f7a169f3e" (UID: "3c8ae678-217a-4501-8cd1-646f7a169f3e"). InnerVolumeSpecName "kube-api-access-bz9d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:52.511837 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:52.511745    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz9d8\" (UniqueName: \"kubernetes.io/projected/3c8ae678-217a-4501-8cd1-646f7a169f3e-kube-api-access-bz9d8\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:56:52.511837 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:52.511785    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8ae678-217a-4501-8cd1-646f7a169f3e-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:56:53.238267 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:53.238240    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp"
Oct 31 03:56:53.238716 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:53.238274    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp" event={"ID":"3c8ae678-217a-4501-8cd1-646f7a169f3e","Type":"ContainerDied","Data":"c74dcd9e582ed4a4d11183be1b5e09172c99a020e71352501f80b29bf6032049"}
Oct 31 03:56:53.238716 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:53.238321    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c74dcd9e582ed4a4d11183be1b5e09172c99a020e71352501f80b29bf6032049"
Oct 31 03:56:53.406667 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:53.406644    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-h526p"
Oct 31 03:56:53.422439 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:53.422416    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edacc995-47b9-4ed7-9a68-7aa8fdce1ce0-kube-api-access-xl2lj\" (UniqueName: \"kubernetes.io/projected/edacc995-47b9-4ed7-9a68-7aa8fdce1ce0-kube-api-access-xl2lj\") pod \"edacc995-47b9-4ed7-9a68-7aa8fdce1ce0\" (UID: \"edacc995-47b9-4ed7-9a68-7aa8fdce1ce0\") "
Oct 31 03:56:53.422569 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:53.422555    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edacc995-47b9-4ed7-9a68-7aa8fdce1ce0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edacc995-47b9-4ed7-9a68-7aa8fdce1ce0-maas-api-xks-ca\") pod \"edacc995-47b9-4ed7-9a68-7aa8fdce1ce0\" (UID: \"edacc995-47b9-4ed7-9a68-7aa8fdce1ce0\") "
Oct 31 03:56:53.422621 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:53.422610    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edacc995-47b9-4ed7-9a68-7aa8fdce1ce0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edacc995-47b9-4ed7-9a68-7aa8fdce1ce0-maas-api-openshift-ca\") pod \"edacc995-47b9-4ed7-9a68-7aa8fdce1ce0\" (UID: \"edacc995-47b9-4ed7-9a68-7aa8fdce1ce0\") "
Oct 31 03:56:53.423105 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:53.423072    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edacc995-47b9-4ed7-9a68-7aa8fdce1ce0-maas-api-openshift-ca" pod "edacc995-47b9-4ed7-9a68-7aa8fdce1ce0" (UID: "edacc995-47b9-4ed7-9a68-7aa8fdce1ce0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:53.424729 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:53.424702    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edacc995-47b9-4ed7-9a68-7aa8fdce1ce0-kube-api-access-xl2lj" pod "edacc995-47b9-4ed7-9a68-7aa8fdce1ce0" (UID: "edacc995-47b9-4ed7-9a68-7aa8fdce1ce0"). InnerVolumeSpecName "kube-api-access-xl2lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:53.424942 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:53.424924    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edacc995-47b9-4ed7-9a68-7aa8fdce1ce0-maas-api-xks-ca" pod "edacc995-47b9-4ed7-9a68-7aa8fdce1ce0" (UID: "edacc995-47b9-4ed7-9a68-7aa8fdce1ce0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:56:53.524618 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:53.524530    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edacc995-47b9-4ed7-9a68-7aa8fdce1ce0-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:56:53.524618 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:53.524565    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edacc995-47b9-4ed7-9a68-7aa8fdce1ce0-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:56:53.524618 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:53.524583    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl2lj\" (UniqueName: \"kubernetes.io/projected/edacc995-47b9-4ed7-9a68-7aa8fdce1ce0-kube-api-access-xl2lj\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:56:54.248293 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:54.248260    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-h526p"
Oct 31 03:56:54.248696 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:54.248296    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-h526p" event={"ID":"edacc995-47b9-4ed7-9a68-7aa8fdce1ce0","Type":"ContainerDied","Data":"4a3711c7fd0ec35479e0d4462b4c196b4abed24450ecf795c6bc6cbb7e6937a9"}
Oct 31 03:56:54.248696 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:56:54.248344    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a3711c7fd0ec35479e0d4462b4c196b4abed24450ecf795c6bc6cbb7e6937a9"
Oct 31 03:57:00.517514 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:00.517479    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5"]
Oct 31 03:57:00.634672 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:00.634623    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg"]
Oct 31 03:57:03.006299 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:03.006261    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn"]
Oct 31 03:57:03.117050 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:03.117012    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx"]
Oct 31 03:57:05.518435 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:05.518369    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5" podUID="dac96790-89ec-43aa-8949-7a8bc510e676" containerName="payload-processing" containerID="cri-o://8152409ed09e0cb8dac3d0ca5237eb2812da56b1f182070b5b0fbf565f38bc73" gracePeriod=25
Oct 31 03:57:05.584786 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:57:05.584755    4447 logging.go:55] [core] [Channel #8615 SubChannel #8617]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.159:50052", ServerName: "10.134.0.159:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.159:50052: connect: connection refused"
Oct 31 03:57:05.635064 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:05.635024    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg" podUID="aebb4eab-1d36-4aad-bcc4-e62ec26db1d4" containerName="payload-pre-processing" containerID="cri-o://ac1b9e64dc4fb83baebeae7b222377b6def0144ad691a3b2a743158f7eb01dfc" gracePeriod=25
Oct 31 03:57:05.820453 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:05.820402    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5"
Oct 31 03:57:05.935168 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:05.935139    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dac96790-89ec-43aa-8949-7a8bc510e676-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac96790-89ec-43aa-8949-7a8bc510e676-plugins-config-volume\") pod \"dac96790-89ec-43aa-8949-7a8bc510e676\" (UID: \"dac96790-89ec-43aa-8949-7a8bc510e676\") "
Oct 31 03:57:05.935526 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:05.935508    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dac96790-89ec-43aa-8949-7a8bc510e676-kube-api-access-c978d\" (UniqueName: \"kubernetes.io/projected/dac96790-89ec-43aa-8949-7a8bc510e676-kube-api-access-c978d\") pod \"dac96790-89ec-43aa-8949-7a8bc510e676\" (UID: \"dac96790-89ec-43aa-8949-7a8bc510e676\") "
Oct 31 03:57:05.935665 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:05.935639    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dac96790-89ec-43aa-8949-7a8bc510e676-plugins-config-volume" pod "dac96790-89ec-43aa-8949-7a8bc510e676" (UID: "dac96790-89ec-43aa-8949-7a8bc510e676"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:57:05.936127 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:05.936103    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac96790-89ec-43aa-8949-7a8bc510e676-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:05.951939 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:05.951879    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac96790-89ec-43aa-8949-7a8bc510e676-kube-api-access-c978d" pod "dac96790-89ec-43aa-8949-7a8bc510e676" (UID: "dac96790-89ec-43aa-8949-7a8bc510e676"). InnerVolumeSpecName "kube-api-access-c978d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:57:05.952323 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:05.952305    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg"
Oct 31 03:57:06.036706 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.036621    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aebb4eab-1d36-4aad-bcc4-e62ec26db1d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aebb4eab-1d36-4aad-bcc4-e62ec26db1d4-plugins-config-volume\") pod \"aebb4eab-1d36-4aad-bcc4-e62ec26db1d4\" (UID: \"aebb4eab-1d36-4aad-bcc4-e62ec26db1d4\") "
Oct 31 03:57:06.036859 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.036750    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aebb4eab-1d36-4aad-bcc4-e62ec26db1d4-kube-api-access-xkdqg\" (UniqueName: \"kubernetes.io/projected/aebb4eab-1d36-4aad-bcc4-e62ec26db1d4-kube-api-access-xkdqg\") pod \"aebb4eab-1d36-4aad-bcc4-e62ec26db1d4\" (UID: \"aebb4eab-1d36-4aad-bcc4-e62ec26db1d4\") "
Oct 31 03:57:06.037087 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.037058    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aebb4eab-1d36-4aad-bcc4-e62ec26db1d4-plugins-config-volume" pod "aebb4eab-1d36-4aad-bcc4-e62ec26db1d4" (UID: "aebb4eab-1d36-4aad-bcc4-e62ec26db1d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:57:06.037358 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.037337    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c978d\" (UniqueName: \"kubernetes.io/projected/dac96790-89ec-43aa-8949-7a8bc510e676-kube-api-access-c978d\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:06.037358 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.037355    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aebb4eab-1d36-4aad-bcc4-e62ec26db1d4-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:06.039068 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.039051    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aebb4eab-1d36-4aad-bcc4-e62ec26db1d4-kube-api-access-xkdqg" pod "aebb4eab-1d36-4aad-bcc4-e62ec26db1d4" (UID: "aebb4eab-1d36-4aad-bcc4-e62ec26db1d4"). InnerVolumeSpecName "kube-api-access-xkdqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:57:06.156321 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.156176    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkdqg\" (UniqueName: \"kubernetes.io/projected/aebb4eab-1d36-4aad-bcc4-e62ec26db1d4-kube-api-access-xkdqg\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:06.411434 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.411341    4447 generic.go:356] "Generic (PLEG): container finished" podID="dac96790-89ec-43aa-8949-7a8bc510e676" containerID="8152409ed09e0cb8dac3d0ca5237eb2812da56b1f182070b5b0fbf565f38bc73" exitCode=0
Oct 31 03:57:06.411620 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.411502    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5"
Oct 31 03:57:06.411620 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.411501    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5" event={"ID":"dac96790-89ec-43aa-8949-7a8bc510e676","Type":"ContainerDied","Data":"8152409ed09e0cb8dac3d0ca5237eb2812da56b1f182070b5b0fbf565f38bc73"}
Oct 31 03:57:06.411747 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.411676    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5" event={"ID":"dac96790-89ec-43aa-8949-7a8bc510e676","Type":"ContainerDied","Data":"8c9fd1116b719c013753cc8e1530646bcd46d8d31d433e550c6f01859fefab34"}
Oct 31 03:57:06.411972 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.411951    4447 scope.go:122] "RemoveContainer" containerID="8152409ed09e0cb8dac3d0ca5237eb2812da56b1f182070b5b0fbf565f38bc73"
Oct 31 03:57:06.415416 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.415386    4447 generic.go:356] "Generic (PLEG): container finished" podID="aebb4eab-1d36-4aad-bcc4-e62ec26db1d4" containerID="ac1b9e64dc4fb83baebeae7b222377b6def0144ad691a3b2a743158f7eb01dfc" exitCode=0
Oct 31 03:57:06.415825 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.415797    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg" event={"ID":"aebb4eab-1d36-4aad-bcc4-e62ec26db1d4","Type":"ContainerDied","Data":"ac1b9e64dc4fb83baebeae7b222377b6def0144ad691a3b2a743158f7eb01dfc"}
Oct 31 03:57:06.416017 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.415879    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg" event={"ID":"aebb4eab-1d36-4aad-bcc4-e62ec26db1d4","Type":"ContainerDied","Data":"e5a44259fe35a097faaf6a5ceb0b222bc3ef0681753332caa6c759a0a0084ca8"}
Oct 31 03:57:06.416093 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.416060    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg"
Oct 31 03:57:06.427790 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.427769    4447 scope.go:122] "RemoveContainer" containerID="8152409ed09e0cb8dac3d0ca5237eb2812da56b1f182070b5b0fbf565f38bc73"
Oct 31 03:57:06.428133 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:57:06.428108    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8152409ed09e0cb8dac3d0ca5237eb2812da56b1f182070b5b0fbf565f38bc73\": container with ID starting with 8152409ed09e0cb8dac3d0ca5237eb2812da56b1f182070b5b0fbf565f38bc73 not found: ID does not exist" containerID="8152409ed09e0cb8dac3d0ca5237eb2812da56b1f182070b5b0fbf565f38bc73"
Oct 31 03:57:06.428202 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.428148    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8152409ed09e0cb8dac3d0ca5237eb2812da56b1f182070b5b0fbf565f38bc73"} err="failed to get container status \"8152409ed09e0cb8dac3d0ca5237eb2812da56b1f182070b5b0fbf565f38bc73\": rpc error: code = NotFound desc = could not find container \"8152409ed09e0cb8dac3d0ca5237eb2812da56b1f182070b5b0fbf565f38bc73\": container with ID starting with 8152409ed09e0cb8dac3d0ca5237eb2812da56b1f182070b5b0fbf565f38bc73 not found: ID does not exist"
Oct 31 03:57:06.428202 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.428173    4447 scope.go:122] "RemoveContainer" containerID="ac1b9e64dc4fb83baebeae7b222377b6def0144ad691a3b2a743158f7eb01dfc"
Oct 31 03:57:06.438406 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.438385    4447 scope.go:122] "RemoveContainer" containerID="ac1b9e64dc4fb83baebeae7b222377b6def0144ad691a3b2a743158f7eb01dfc"
Oct 31 03:57:06.438686 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:57:06.438665    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1b9e64dc4fb83baebeae7b222377b6def0144ad691a3b2a743158f7eb01dfc\": container with ID starting with ac1b9e64dc4fb83baebeae7b222377b6def0144ad691a3b2a743158f7eb01dfc not found: ID does not exist" containerID="ac1b9e64dc4fb83baebeae7b222377b6def0144ad691a3b2a743158f7eb01dfc"
Oct 31 03:57:06.438753 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.438694    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1b9e64dc4fb83baebeae7b222377b6def0144ad691a3b2a743158f7eb01dfc"} err="failed to get container status \"ac1b9e64dc4fb83baebeae7b222377b6def0144ad691a3b2a743158f7eb01dfc\": rpc error: code = NotFound desc = could not find container \"ac1b9e64dc4fb83baebeae7b222377b6def0144ad691a3b2a743158f7eb01dfc\": container with ID starting with ac1b9e64dc4fb83baebeae7b222377b6def0144ad691a3b2a743158f7eb01dfc not found: ID does not exist"
Oct 31 03:57:06.579963 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.579644    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg"]
Oct 31 03:57:06.584157 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.584134    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1180d818-588986988c-4z5dg"]
Oct 31 03:57:06.585196 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.585172    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5" podUID="dac96790-89ec-43aa-8949-7a8bc510e676" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.159:50052\" within 1s: context deadline exceeded"
Oct 31 03:57:06.609613 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.609582    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aebb4eab-1d36-4aad-bcc4-e62ec26db1d4" path="/var/lib/kubelet/pods/aebb4eab-1d36-4aad-bcc4-e62ec26db1d4/volumes"
Oct 31 03:57:06.693321 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.693244    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5"]
Oct 31 03:57:06.700675 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:06.700646    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1180d818-5c665fb697-5vzh5"]
Oct 31 03:57:08.011350 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.006982    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn" podUID="8b5f34fa-2f2e-4fe0-b77f-02667175ac6a" containerName="payload-processing" containerID="cri-o://6ec7952c37e52cd280e3438bed64d3ef1e1eb215cd1bb25df091e473d495019d" gracePeriod=25
Oct 31 03:57:08.118522 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.118484    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx" podUID="6afd09e9-4640-4fb7-87a3-1cc053929b2c" containerName="payload-pre-processing" containerID="cri-o://c15e5507bbbe11a8601252b30f00e8e81777356cc8d8102b57224602a8fff97a" gracePeriod=25
Oct 31 03:57:08.132926 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.132879    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph"]
Oct 31 03:57:08.180910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.180851    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c43a6-h5hph"]
Oct 31 03:57:08.412636 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.412609    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn"
Oct 31 03:57:08.437035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.435517    4447 generic.go:356] "Generic (PLEG): container finished" podID="8b5f34fa-2f2e-4fe0-b77f-02667175ac6a" containerID="6ec7952c37e52cd280e3438bed64d3ef1e1eb215cd1bb25df091e473d495019d" exitCode=0
Oct 31 03:57:08.437035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.435745    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn" event={"ID":"8b5f34fa-2f2e-4fe0-b77f-02667175ac6a","Type":"ContainerDied","Data":"6ec7952c37e52cd280e3438bed64d3ef1e1eb215cd1bb25df091e473d495019d"}
Oct 31 03:57:08.437035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.435796    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn" event={"ID":"8b5f34fa-2f2e-4fe0-b77f-02667175ac6a","Type":"ContainerDied","Data":"f91b5da5bf7e7b5c1f96846bf3a5fdb8771e79fb958bbcc218c1e609367ee00b"}
Oct 31 03:57:08.437035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.435827    4447 scope.go:122] "RemoveContainer" containerID="6ec7952c37e52cd280e3438bed64d3ef1e1eb215cd1bb25df091e473d495019d"
Oct 31 03:57:08.437035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.435991    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn"
Oct 31 03:57:08.437917 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.437862    4447 generic.go:356] "Generic (PLEG): container finished" podID="6afd09e9-4640-4fb7-87a3-1cc053929b2c" containerID="c15e5507bbbe11a8601252b30f00e8e81777356cc8d8102b57224602a8fff97a" exitCode=0
Oct 31 03:57:08.438014 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.437933    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx" event={"ID":"6afd09e9-4640-4fb7-87a3-1cc053929b2c","Type":"ContainerDied","Data":"c15e5507bbbe11a8601252b30f00e8e81777356cc8d8102b57224602a8fff97a"}
Oct 31 03:57:08.438014 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.437989    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx" event={"ID":"6afd09e9-4640-4fb7-87a3-1cc053929b2c","Type":"ContainerDied","Data":"5ff9dbbcddf7b518cc25b21c20c7ccfc7d898442a8ef9b70074fee37a07c71ea"}
Oct 31 03:57:08.438092 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.438016    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ff9dbbcddf7b518cc25b21c20c7ccfc7d898442a8ef9b70074fee37a07c71ea"
Oct 31 03:57:08.451155 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.450708    4447 scope.go:122] "RemoveContainer" containerID="6ec7952c37e52cd280e3438bed64d3ef1e1eb215cd1bb25df091e473d495019d"
Oct 31 03:57:08.451379 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:57:08.451307    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ec7952c37e52cd280e3438bed64d3ef1e1eb215cd1bb25df091e473d495019d\": container with ID starting with 6ec7952c37e52cd280e3438bed64d3ef1e1eb215cd1bb25df091e473d495019d not found: ID does not exist" containerID="6ec7952c37e52cd280e3438bed64d3ef1e1eb215cd1bb25df091e473d495019d"
Oct 31 03:57:08.451379 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.451342    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ec7952c37e52cd280e3438bed64d3ef1e1eb215cd1bb25df091e473d495019d"} err="failed to get container status \"6ec7952c37e52cd280e3438bed64d3ef1e1eb215cd1bb25df091e473d495019d\": rpc error: code = NotFound desc = could not find container \"6ec7952c37e52cd280e3438bed64d3ef1e1eb215cd1bb25df091e473d495019d\": container with ID starting with 6ec7952c37e52cd280e3438bed64d3ef1e1eb215cd1bb25df091e473d495019d not found: ID does not exist"
Oct 31 03:57:08.452965 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.452943    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx"
Oct 31 03:57:08.492128 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.492095    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b5f34fa-2f2e-4fe0-b77f-02667175ac6a-kube-api-access-sclkf\" (UniqueName: \"kubernetes.io/projected/8b5f34fa-2f2e-4fe0-b77f-02667175ac6a-kube-api-access-sclkf\") pod \"8b5f34fa-2f2e-4fe0-b77f-02667175ac6a\" (UID: \"8b5f34fa-2f2e-4fe0-b77f-02667175ac6a\") "
Oct 31 03:57:08.492361 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.492344    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b5f34fa-2f2e-4fe0-b77f-02667175ac6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5f34fa-2f2e-4fe0-b77f-02667175ac6a-plugins-config-volume\") pod \"8b5f34fa-2f2e-4fe0-b77f-02667175ac6a\" (UID: \"8b5f34fa-2f2e-4fe0-b77f-02667175ac6a\") "
Oct 31 03:57:08.492840 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.492803    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b5f34fa-2f2e-4fe0-b77f-02667175ac6a-plugins-config-volume" pod "8b5f34fa-2f2e-4fe0-b77f-02667175ac6a" (UID: "8b5f34fa-2f2e-4fe0-b77f-02667175ac6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:57:08.495041 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.495019    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b5f34fa-2f2e-4fe0-b77f-02667175ac6a-kube-api-access-sclkf" pod "8b5f34fa-2f2e-4fe0-b77f-02667175ac6a" (UID: "8b5f34fa-2f2e-4fe0-b77f-02667175ac6a"). InnerVolumeSpecName "kube-api-access-sclkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:57:08.593954 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.593861    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6afd09e9-4640-4fb7-87a3-1cc053929b2c-kube-api-access-hggv5\" (UniqueName: \"kubernetes.io/projected/6afd09e9-4640-4fb7-87a3-1cc053929b2c-kube-api-access-hggv5\") pod \"6afd09e9-4640-4fb7-87a3-1cc053929b2c\" (UID: \"6afd09e9-4640-4fb7-87a3-1cc053929b2c\") "
Oct 31 03:57:08.594130 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.594001    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6afd09e9-4640-4fb7-87a3-1cc053929b2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afd09e9-4640-4fb7-87a3-1cc053929b2c-plugins-config-volume\") pod \"6afd09e9-4640-4fb7-87a3-1cc053929b2c\" (UID: \"6afd09e9-4640-4fb7-87a3-1cc053929b2c\") "
Oct 31 03:57:08.594343 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.594319    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afd09e9-4640-4fb7-87a3-1cc053929b2c-plugins-config-volume" pod "6afd09e9-4640-4fb7-87a3-1cc053929b2c" (UID: "6afd09e9-4640-4fb7-87a3-1cc053929b2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:57:08.594591 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.594579    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5f34fa-2f2e-4fe0-b77f-02667175ac6a-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:08.594638 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.594594    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sclkf\" (UniqueName: \"kubernetes.io/projected/8b5f34fa-2f2e-4fe0-b77f-02667175ac6a-kube-api-access-sclkf\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:08.594638 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.594602    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afd09e9-4640-4fb7-87a3-1cc053929b2c-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:08.596104 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.596086    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6afd09e9-4640-4fb7-87a3-1cc053929b2c-kube-api-access-hggv5" pod "6afd09e9-4640-4fb7-87a3-1cc053929b2c" (UID: "6afd09e9-4640-4fb7-87a3-1cc053929b2c"). InnerVolumeSpecName "kube-api-access-hggv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:57:08.610690 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.610660    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c8fd020-89d4-4f2b-93c1-115a0d20c7fd" path="/var/lib/kubelet/pods/5c8fd020-89d4-4f2b-93c1-115a0d20c7fd/volumes"
Oct 31 03:57:08.611171 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.611157    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dac96790-89ec-43aa-8949-7a8bc510e676" path="/var/lib/kubelet/pods/dac96790-89ec-43aa-8949-7a8bc510e676/volumes"
Oct 31 03:57:08.696568 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.696535    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hggv5\" (UniqueName: \"kubernetes.io/projected/6afd09e9-4640-4fb7-87a3-1cc053929b2c-kube-api-access-hggv5\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:08.854518 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.854435    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn"]
Oct 31 03:57:08.858863 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:08.858836    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e02dd5-779459897c-5gmbn"]
Oct 31 03:57:09.446289 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:09.446262    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx"
Oct 31 03:57:09.613840 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:09.613807    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx"]
Oct 31 03:57:09.663160 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:09.663123    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e02dd5-5bfc8bc9ff-cw2zx"]
Oct 31 03:57:10.610551 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:10.610507    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6afd09e9-4640-4fb7-87a3-1cc053929b2c" path="/var/lib/kubelet/pods/6afd09e9-4640-4fb7-87a3-1cc053929b2c/volumes"
Oct 31 03:57:10.611165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:10.611144    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b5f34fa-2f2e-4fe0-b77f-02667175ac6a" path="/var/lib/kubelet/pods/8b5f34fa-2f2e-4fe0-b77f-02667175ac6a/volumes"
Oct 31 03:57:15.713066 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.713028    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c"]
Oct 31 03:57:15.713768 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.713753    4447 cpu_manager.go:414] "containerMap: removing container" podUID="8b5f34fa-2f2e-4fe0-b77f-02667175ac6a" containerName="payload-processing"
Oct 31 03:57:15.713817 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.713769    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b5f34fa-2f2e-4fe0-b77f-02667175ac6a" containerName="payload-processing"
Oct 31 03:57:15.713817 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.713789    4447 cpu_manager.go:414] "containerMap: removing container" podUID="edacc995-47b9-4ed7-9a68-7aa8fdce1ce0" containerName="revoke-keys"
Oct 31 03:57:15.713817 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.713794    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edacc995-47b9-4ed7-9a68-7aa8fdce1ce0" containerName="revoke-keys"
Oct 31 03:57:15.713817 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.713804    4447 cpu_manager.go:414] "containerMap: removing container" podUID="3c8ae678-217a-4501-8cd1-646f7a169f3e" containerName="revoke-keys"
Oct 31 03:57:15.713817 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.713809    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c8ae678-217a-4501-8cd1-646f7a169f3e" containerName="revoke-keys"
Oct 31 03:57:15.713817 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.713819    4447 cpu_manager.go:414] "containerMap: removing container" podUID="6afd09e9-4640-4fb7-87a3-1cc053929b2c" containerName="payload-pre-processing"
Oct 31 03:57:15.714035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.713824    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6afd09e9-4640-4fb7-87a3-1cc053929b2c" containerName="payload-pre-processing"
Oct 31 03:57:15.714035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.713838    4447 cpu_manager.go:414] "containerMap: removing container" podUID="dac96790-89ec-43aa-8949-7a8bc510e676" containerName="payload-processing"
Oct 31 03:57:15.714035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.713844    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dac96790-89ec-43aa-8949-7a8bc510e676" containerName="payload-processing"
Oct 31 03:57:15.714035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.713854    4447 cpu_manager.go:414] "containerMap: removing container" podUID="aebb4eab-1d36-4aad-bcc4-e62ec26db1d4" containerName="payload-pre-processing"
Oct 31 03:57:15.714035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.713859    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aebb4eab-1d36-4aad-bcc4-e62ec26db1d4" containerName="payload-pre-processing"
Oct 31 03:57:15.714035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.713983    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="aebb4eab-1d36-4aad-bcc4-e62ec26db1d4" containerName="payload-pre-processing"
Oct 31 03:57:15.714035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.713999    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b5f34fa-2f2e-4fe0-b77f-02667175ac6a" containerName="payload-processing"
Oct 31 03:57:15.714035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.714006    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="edacc995-47b9-4ed7-9a68-7aa8fdce1ce0" containerName="revoke-keys"
Oct 31 03:57:15.714035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.714011    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c8ae678-217a-4501-8cd1-646f7a169f3e" containerName="revoke-keys"
Oct 31 03:57:15.714035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.714017    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="6afd09e9-4640-4fb7-87a3-1cc053929b2c" containerName="payload-pre-processing"
Oct 31 03:57:15.714035 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.714022    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="dac96790-89ec-43aa-8949-7a8bc510e676" containerName="payload-processing"
Oct 31 03:57:15.719063 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.719046    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c"
Oct 31 03:57:15.776528 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.776494    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c"]
Oct 31 03:57:15.803571 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.803535    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j6pj\" (UniqueName: \"kubernetes.io/projected/648d0647-a65a-4d19-b539-b52e5cfc9cf8-kube-api-access-6j6pj\") pod \"maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c\" (UID: \"648d0647-a65a-4d19-b539-b52e5cfc9cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c"
Oct 31 03:57:15.803743 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.803592    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648d0647-a65a-4d19-b539-b52e5cfc9cf8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c\" (UID: \"648d0647-a65a-4d19-b539-b52e5cfc9cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c"
Oct 31 03:57:15.803786 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.803777    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648d0647-a65a-4d19-b539-b52e5cfc9cf8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c\" (UID: \"648d0647-a65a-4d19-b539-b52e5cfc9cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c"
Oct 31 03:57:15.820410 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.820382    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:57:15.820571 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.820465    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 03:57:15.906059 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.905932    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648d0647-a65a-4d19-b539-b52e5cfc9cf8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c\" (UID: \"648d0647-a65a-4d19-b539-b52e5cfc9cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c"
Oct 31 03:57:15.906253 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.906072    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j6pj\" (UniqueName: \"kubernetes.io/projected/648d0647-a65a-4d19-b539-b52e5cfc9cf8-kube-api-access-6j6pj\") pod \"maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c\" (UID: \"648d0647-a65a-4d19-b539-b52e5cfc9cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c"
Oct 31 03:57:15.906253 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.906139    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648d0647-a65a-4d19-b539-b52e5cfc9cf8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c\" (UID: \"648d0647-a65a-4d19-b539-b52e5cfc9cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c"
Oct 31 03:57:15.906769 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.906747    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648d0647-a65a-4d19-b539-b52e5cfc9cf8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c\" (UID: \"648d0647-a65a-4d19-b539-b52e5cfc9cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c"
Oct 31 03:57:15.908491 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.908468    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648d0647-a65a-4d19-b539-b52e5cfc9cf8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c\" (UID: \"648d0647-a65a-4d19-b539-b52e5cfc9cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c"
Oct 31 03:57:15.967293 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:15.967220    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j6pj\" (UniqueName: \"kubernetes.io/projected/648d0647-a65a-4d19-b539-b52e5cfc9cf8-kube-api-access-6j6pj\") pod \"maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c\" (UID: \"648d0647-a65a-4d19-b539-b52e5cfc9cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c"
Oct 31 03:57:16.033467 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:16.033431    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c"
Oct 31 03:57:16.426396 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:57:16.426363    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648d0647_a65a_4d19_b539_b52e5cfc9cf8.slice/crio-17489e092bddffec46f752d77a83329d0b4f24be991eb7bd6e91a36ff03c3044 WatchSource:0}: Error finding container 17489e092bddffec46f752d77a83329d0b4f24be991eb7bd6e91a36ff03c3044: Status 404 returned error can't find the container with id 17489e092bddffec46f752d77a83329d0b4f24be991eb7bd6e91a36ff03c3044
Oct 31 03:57:16.426541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:16.426470    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c"]
Oct 31 03:57:16.504952 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:16.504918    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c" event={"ID":"648d0647-a65a-4d19-b539-b52e5cfc9cf8","Type":"ContainerStarted","Data":"17489e092bddffec46f752d77a83329d0b4f24be991eb7bd6e91a36ff03c3044"}
Oct 31 03:57:17.515032 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:17.514999    4447 generic.go:356] "Generic (PLEG): container finished" podID="648d0647-a65a-4d19-b539-b52e5cfc9cf8" containerID="86dbd2ed7737cbd0d6b4d3509e7ce9f208c04c27ddc980cb3574d6a8dc3a8603" exitCode=0
Oct 31 03:57:17.515442 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:17.515083    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c" event={"ID":"648d0647-a65a-4d19-b539-b52e5cfc9cf8","Type":"ContainerDied","Data":"86dbd2ed7737cbd0d6b4d3509e7ce9f208c04c27ddc980cb3574d6a8dc3a8603"}
Oct 31 03:57:18.687243 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:18.687218    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c"
Oct 31 03:57:18.752452 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:18.752416    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/648d0647-a65a-4d19-b539-b52e5cfc9cf8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648d0647-a65a-4d19-b539-b52e5cfc9cf8-maas-api-xks-ca\") pod \"648d0647-a65a-4d19-b539-b52e5cfc9cf8\" (UID: \"648d0647-a65a-4d19-b539-b52e5cfc9cf8\") "
Oct 31 03:57:18.752659 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:18.752509    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/648d0647-a65a-4d19-b539-b52e5cfc9cf8-kube-api-access-6j6pj\" (UniqueName: \"kubernetes.io/projected/648d0647-a65a-4d19-b539-b52e5cfc9cf8-kube-api-access-6j6pj\") pod \"648d0647-a65a-4d19-b539-b52e5cfc9cf8\" (UID: \"648d0647-a65a-4d19-b539-b52e5cfc9cf8\") "
Oct 31 03:57:18.752731 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:18.752657    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/648d0647-a65a-4d19-b539-b52e5cfc9cf8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648d0647-a65a-4d19-b539-b52e5cfc9cf8-maas-api-openshift-ca\") pod \"648d0647-a65a-4d19-b539-b52e5cfc9cf8\" (UID: \"648d0647-a65a-4d19-b539-b52e5cfc9cf8\") "
Oct 31 03:57:18.753098 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:18.753071    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/648d0647-a65a-4d19-b539-b52e5cfc9cf8-maas-api-openshift-ca" pod "648d0647-a65a-4d19-b539-b52e5cfc9cf8" (UID: "648d0647-a65a-4d19-b539-b52e5cfc9cf8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:57:18.753334 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:18.753318    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648d0647-a65a-4d19-b539-b52e5cfc9cf8-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:18.754690 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:18.754669    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/648d0647-a65a-4d19-b539-b52e5cfc9cf8-maas-api-xks-ca" pod "648d0647-a65a-4d19-b539-b52e5cfc9cf8" (UID: "648d0647-a65a-4d19-b539-b52e5cfc9cf8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:57:18.754783 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:18.754757    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648d0647-a65a-4d19-b539-b52e5cfc9cf8-kube-api-access-6j6pj" pod "648d0647-a65a-4d19-b539-b52e5cfc9cf8" (UID: "648d0647-a65a-4d19-b539-b52e5cfc9cf8"). InnerVolumeSpecName "kube-api-access-6j6pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:57:18.855037 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:18.854958    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648d0647-a65a-4d19-b539-b52e5cfc9cf8-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:18.855037 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:18.854989    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j6pj\" (UniqueName: \"kubernetes.io/projected/648d0647-a65a-4d19-b539-b52e5cfc9cf8-kube-api-access-6j6pj\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:19.004621 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:57:19.004582    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:57:19.533215 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:19.533187    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c"
Oct 31 03:57:19.533215 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:19.533194    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c" event={"ID":"648d0647-a65a-4d19-b539-b52e5cfc9cf8","Type":"ContainerDied","Data":"17489e092bddffec46f752d77a83329d0b4f24be991eb7bd6e91a36ff03c3044"}
Oct 31 03:57:19.533445 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:19.533270    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17489e092bddffec46f752d77a83329d0b4f24be991eb7bd6e91a36ff03c3044"
Oct 31 03:57:27.206381 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:27.206345    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx"]
Oct 31 03:57:27.377583 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:27.377542    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh"]
Oct 31 03:57:29.334201 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:29.334164    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt"]
Oct 31 03:57:29.384188 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:29.384153    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-36664b-4k2qt"]
Oct 31 03:57:30.611291 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:30.611261    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6d924e9-f7d4-435f-b07b-68263ee2ab69" path="/var/lib/kubelet/pods/c6d924e9-f7d4-435f-b07b-68263ee2ab69/volumes"
Oct 31 03:57:32.206872 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.206835    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx" podUID="bfb6def7-2ef9-4cf2-974f-dc8f012eb431" containerName="payload-processing" containerID="cri-o://cda7102f07320ac52b36e40a7e7dd594bae6acb8d2b46dad1dc60da2a6777ed4" gracePeriod=25
Oct 31 03:57:32.378814 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.378764    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh" podUID="55f914e0-b8b7-403b-8ad0-1e4ef8d348dd" containerName="payload-pre-processing" containerID="cri-o://382e7568fab302ffdad09bfe207ae500bf55d1fd400cccd3a707e3d37c2b18bb" gracePeriod=25
Oct 31 03:57:32.516166 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.516142    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx"
Oct 31 03:57:32.648253 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.648215    4447 generic.go:356] "Generic (PLEG): container finished" podID="55f914e0-b8b7-403b-8ad0-1e4ef8d348dd" containerID="382e7568fab302ffdad09bfe207ae500bf55d1fd400cccd3a707e3d37c2b18bb" exitCode=0
Oct 31 03:57:32.648428 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.648289    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh" event={"ID":"55f914e0-b8b7-403b-8ad0-1e4ef8d348dd","Type":"ContainerDied","Data":"382e7568fab302ffdad09bfe207ae500bf55d1fd400cccd3a707e3d37c2b18bb"}
Oct 31 03:57:32.650041 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.650013    4447 generic.go:356] "Generic (PLEG): container finished" podID="bfb6def7-2ef9-4cf2-974f-dc8f012eb431" containerID="cda7102f07320ac52b36e40a7e7dd594bae6acb8d2b46dad1dc60da2a6777ed4" exitCode=0
Oct 31 03:57:32.650193 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.650050    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx" event={"ID":"bfb6def7-2ef9-4cf2-974f-dc8f012eb431","Type":"ContainerDied","Data":"cda7102f07320ac52b36e40a7e7dd594bae6acb8d2b46dad1dc60da2a6777ed4"}
Oct 31 03:57:32.650193 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.650097    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx"
Oct 31 03:57:32.650193 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.650112    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx" event={"ID":"bfb6def7-2ef9-4cf2-974f-dc8f012eb431","Type":"ContainerDied","Data":"c5ea9acb14883e4b9dedd5ad461f625cee33fed6cc208eff3067a2c93ee181c8"}
Oct 31 03:57:32.650193 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.650143    4447 scope.go:122] "RemoveContainer" containerID="cda7102f07320ac52b36e40a7e7dd594bae6acb8d2b46dad1dc60da2a6777ed4"
Oct 31 03:57:32.651539 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.651521    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh"
Oct 31 03:57:32.664205 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.664171    4447 scope.go:122] "RemoveContainer" containerID="cda7102f07320ac52b36e40a7e7dd594bae6acb8d2b46dad1dc60da2a6777ed4"
Oct 31 03:57:32.666935 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:57:32.664566    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda7102f07320ac52b36e40a7e7dd594bae6acb8d2b46dad1dc60da2a6777ed4\": container with ID starting with cda7102f07320ac52b36e40a7e7dd594bae6acb8d2b46dad1dc60da2a6777ed4 not found: ID does not exist" containerID="cda7102f07320ac52b36e40a7e7dd594bae6acb8d2b46dad1dc60da2a6777ed4"
Oct 31 03:57:32.666935 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.664608    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda7102f07320ac52b36e40a7e7dd594bae6acb8d2b46dad1dc60da2a6777ed4"} err="failed to get container status \"cda7102f07320ac52b36e40a7e7dd594bae6acb8d2b46dad1dc60da2a6777ed4\": rpc error: code = NotFound desc = could not find container \"cda7102f07320ac52b36e40a7e7dd594bae6acb8d2b46dad1dc60da2a6777ed4\": container with ID starting with cda7102f07320ac52b36e40a7e7dd594bae6acb8d2b46dad1dc60da2a6777ed4 not found: ID does not exist"
Oct 31 03:57:32.678724 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.678691    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfb6def7-2ef9-4cf2-974f-dc8f012eb431-kube-api-access-vrn62\" (UniqueName: \"kubernetes.io/projected/bfb6def7-2ef9-4cf2-974f-dc8f012eb431-kube-api-access-vrn62\") pod \"bfb6def7-2ef9-4cf2-974f-dc8f012eb431\" (UID: \"bfb6def7-2ef9-4cf2-974f-dc8f012eb431\") "
Oct 31 03:57:32.678953 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.678931    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfb6def7-2ef9-4cf2-974f-dc8f012eb431-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb6def7-2ef9-4cf2-974f-dc8f012eb431-plugins-config-volume\") pod \"bfb6def7-2ef9-4cf2-974f-dc8f012eb431\" (UID: \"bfb6def7-2ef9-4cf2-974f-dc8f012eb431\") "
Oct 31 03:57:32.680387 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.680357    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb6def7-2ef9-4cf2-974f-dc8f012eb431-plugins-config-volume" pod "bfb6def7-2ef9-4cf2-974f-dc8f012eb431" (UID: "bfb6def7-2ef9-4cf2-974f-dc8f012eb431"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:57:32.682603 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.682571    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb6def7-2ef9-4cf2-974f-dc8f012eb431-kube-api-access-vrn62" pod "bfb6def7-2ef9-4cf2-974f-dc8f012eb431" (UID: "bfb6def7-2ef9-4cf2-974f-dc8f012eb431"). InnerVolumeSpecName "kube-api-access-vrn62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:57:32.780477 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.780399    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f914e0-b8b7-403b-8ad0-1e4ef8d348dd-kube-api-access-sm7kn\" (UniqueName: \"kubernetes.io/projected/55f914e0-b8b7-403b-8ad0-1e4ef8d348dd-kube-api-access-sm7kn\") pod \"55f914e0-b8b7-403b-8ad0-1e4ef8d348dd\" (UID: \"55f914e0-b8b7-403b-8ad0-1e4ef8d348dd\") "
Oct 31 03:57:32.780628 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.780597    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f914e0-b8b7-403b-8ad0-1e4ef8d348dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55f914e0-b8b7-403b-8ad0-1e4ef8d348dd-plugins-config-volume\") pod \"55f914e0-b8b7-403b-8ad0-1e4ef8d348dd\" (UID: \"55f914e0-b8b7-403b-8ad0-1e4ef8d348dd\") "
Oct 31 03:57:32.781069 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.781041    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f914e0-b8b7-403b-8ad0-1e4ef8d348dd-plugins-config-volume" pod "55f914e0-b8b7-403b-8ad0-1e4ef8d348dd" (UID: "55f914e0-b8b7-403b-8ad0-1e4ef8d348dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:57:32.781314 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.781294    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrn62\" (UniqueName: \"kubernetes.io/projected/bfb6def7-2ef9-4cf2-974f-dc8f012eb431-kube-api-access-vrn62\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:32.781314 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.781316    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55f914e0-b8b7-403b-8ad0-1e4ef8d348dd-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:32.781437 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.781325    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb6def7-2ef9-4cf2-974f-dc8f012eb431-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:32.782695 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.782669    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f914e0-b8b7-403b-8ad0-1e4ef8d348dd-kube-api-access-sm7kn" pod "55f914e0-b8b7-403b-8ad0-1e4ef8d348dd" (UID: "55f914e0-b8b7-403b-8ad0-1e4ef8d348dd"). InnerVolumeSpecName "kube-api-access-sm7kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:57:32.882532 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:32.882499    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm7kn\" (UniqueName: \"kubernetes.io/projected/55f914e0-b8b7-403b-8ad0-1e4ef8d348dd-kube-api-access-sm7kn\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:33.073152 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:33.073044    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx"]
Oct 31 03:57:33.077336 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:33.077307    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-05a9ff-5b8d678595-k8znx"]
Oct 31 03:57:33.660193 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:33.660126    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh"
Oct 31 03:57:33.660193 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:33.660133    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh" event={"ID":"55f914e0-b8b7-403b-8ad0-1e4ef8d348dd","Type":"ContainerDied","Data":"311e0798b5a08fcfb4e52a9899510c289dbc09787a46490a36669d1a769598c3"}
Oct 31 03:57:33.661113 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:33.660203    4447 scope.go:122] "RemoveContainer" containerID="382e7568fab302ffdad09bfe207ae500bf55d1fd400cccd3a707e3d37c2b18bb"
Oct 31 03:57:33.711720 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:57:33.711692    4447 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f914e0_b8b7_403b_8ad0_1e4ef8d348dd.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f914e0_b8b7_403b_8ad0_1e4ef8d348dd.slice/crio-311e0798b5a08fcfb4e52a9899510c289dbc09787a46490a36669d1a769598c3\": RecentStats: unable to find data in memory cache]"
Oct 31 03:57:33.827084 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:33.827054    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh"]
Oct 31 03:57:33.834054 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:33.834015    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-05a9ff-57b87c79b7-7brfh"]
Oct 31 03:57:34.610283 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:34.610253    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f914e0-b8b7-403b-8ad0-1e4ef8d348dd" path="/var/lib/kubelet/pods/55f914e0-b8b7-403b-8ad0-1e4ef8d348dd/volumes"
Oct 31 03:57:34.610772 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:34.610752    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfb6def7-2ef9-4cf2-974f-dc8f012eb431" path="/var/lib/kubelet/pods/bfb6def7-2ef9-4cf2-974f-dc8f012eb431/volumes"
Oct 31 03:57:36.630261 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:36.630228    4447 scope.go:122] "RemoveContainer" containerID="e01834ac92a67c31a52a687ce79fa769f05c74d7045a1e4a788265c4d09759f5"
Oct 31 03:57:36.911271 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:36.911194    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h"]
Oct 31 03:57:36.911931 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:36.911910    4447 cpu_manager.go:414] "containerMap: removing container" podUID="648d0647-a65a-4d19-b539-b52e5cfc9cf8" containerName="revoke-keys"
Oct 31 03:57:36.911982 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:36.911933    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="648d0647-a65a-4d19-b539-b52e5cfc9cf8" containerName="revoke-keys"
Oct 31 03:57:36.911982 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:36.911948    4447 cpu_manager.go:414] "containerMap: removing container" podUID="bfb6def7-2ef9-4cf2-974f-dc8f012eb431" containerName="payload-processing"
Oct 31 03:57:36.911982 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:36.911954    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfb6def7-2ef9-4cf2-974f-dc8f012eb431" containerName="payload-processing"
Oct 31 03:57:36.911982 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:36.911969    4447 cpu_manager.go:414] "containerMap: removing container" podUID="55f914e0-b8b7-403b-8ad0-1e4ef8d348dd" containerName="payload-pre-processing"
Oct 31 03:57:36.911982 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:36.911975    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f914e0-b8b7-403b-8ad0-1e4ef8d348dd" containerName="payload-pre-processing"
Oct 31 03:57:36.912138 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:36.912086    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="648d0647-a65a-4d19-b539-b52e5cfc9cf8" containerName="revoke-keys"
Oct 31 03:57:36.912138 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:36.912095    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfb6def7-2ef9-4cf2-974f-dc8f012eb431" containerName="payload-processing"
Oct 31 03:57:36.912138 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:36.912101    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f914e0-b8b7-403b-8ad0-1e4ef8d348dd" containerName="payload-pre-processing"
Oct 31 03:57:36.917537 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:36.917518    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h"
Oct 31 03:57:36.938730 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:36.938694    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f76a9936-58a5-4203-a1c8-9d8d51d3a190-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h\" (UID: \"f76a9936-58a5-4203-a1c8-9d8d51d3a190\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h"
Oct 31 03:57:36.938867 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:36.938766    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f76a9936-58a5-4203-a1c8-9d8d51d3a190-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h\" (UID: \"f76a9936-58a5-4203-a1c8-9d8d51d3a190\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h"
Oct 31 03:57:36.939029 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:36.939006    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfx7r\" (UniqueName: \"kubernetes.io/projected/f76a9936-58a5-4203-a1c8-9d8d51d3a190-kube-api-access-nfx7r\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h\" (UID: \"f76a9936-58a5-4203-a1c8-9d8d51d3a190\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h"
Oct 31 03:57:36.974371 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:36.974334    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h"]
Oct 31 03:57:37.018696 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:37.018661    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:57:37.018943 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:37.018924    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 03:57:37.040962 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:37.040929    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfx7r\" (UniqueName: \"kubernetes.io/projected/f76a9936-58a5-4203-a1c8-9d8d51d3a190-kube-api-access-nfx7r\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h\" (UID: \"f76a9936-58a5-4203-a1c8-9d8d51d3a190\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h"
Oct 31 03:57:37.041155 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:37.041011    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f76a9936-58a5-4203-a1c8-9d8d51d3a190-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h\" (UID: \"f76a9936-58a5-4203-a1c8-9d8d51d3a190\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h"
Oct 31 03:57:37.041155 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:37.041039    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f76a9936-58a5-4203-a1c8-9d8d51d3a190-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h\" (UID: \"f76a9936-58a5-4203-a1c8-9d8d51d3a190\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h"
Oct 31 03:57:37.041686 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:37.041662    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f76a9936-58a5-4203-a1c8-9d8d51d3a190-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h\" (UID: \"f76a9936-58a5-4203-a1c8-9d8d51d3a190\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h"
Oct 31 03:57:37.043366 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:37.043348    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f76a9936-58a5-4203-a1c8-9d8d51d3a190-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h\" (UID: \"f76a9936-58a5-4203-a1c8-9d8d51d3a190\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h"
Oct 31 03:57:37.102862 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:37.102821    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfx7r\" (UniqueName: \"kubernetes.io/projected/f76a9936-58a5-4203-a1c8-9d8d51d3a190-kube-api-access-nfx7r\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h\" (UID: \"f76a9936-58a5-4203-a1c8-9d8d51d3a190\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h"
Oct 31 03:57:37.231403 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:37.231370    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h"
Oct 31 03:57:37.617838 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:37.617807    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h"]
Oct 31 03:57:37.621507 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:57:37.621478    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76a9936_58a5_4203_a1c8_9d8d51d3a190.slice/crio-ee46e9c17e03cebe244d734b742251d48e20e0355a72a83b0ef90e87fc4dfa27 WatchSource:0}: Error finding container ee46e9c17e03cebe244d734b742251d48e20e0355a72a83b0ef90e87fc4dfa27: Status 404 returned error can't find the container with id ee46e9c17e03cebe244d734b742251d48e20e0355a72a83b0ef90e87fc4dfa27
Oct 31 03:57:37.698691 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:37.698658    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h" event={"ID":"f76a9936-58a5-4203-a1c8-9d8d51d3a190","Type":"ContainerStarted","Data":"ee46e9c17e03cebe244d734b742251d48e20e0355a72a83b0ef90e87fc4dfa27"}
Oct 31 03:57:38.707906 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:38.707853    4447 generic.go:356] "Generic (PLEG): container finished" podID="f76a9936-58a5-4203-a1c8-9d8d51d3a190" containerID="d20f80c557413dfaf56c9dd8c9cc9d48b10efde114bf4bef9b289f71fb0c1151" exitCode=0
Oct 31 03:57:38.708314 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:38.707948    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h" event={"ID":"f76a9936-58a5-4203-a1c8-9d8d51d3a190","Type":"ContainerDied","Data":"d20f80c557413dfaf56c9dd8c9cc9d48b10efde114bf4bef9b289f71fb0c1151"}
Oct 31 03:57:39.893558 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:39.893532    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h"
Oct 31 03:57:39.983994 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:39.983873    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f76a9936-58a5-4203-a1c8-9d8d51d3a190-kube-api-access-nfx7r\" (UniqueName: \"kubernetes.io/projected/f76a9936-58a5-4203-a1c8-9d8d51d3a190-kube-api-access-nfx7r\") pod \"f76a9936-58a5-4203-a1c8-9d8d51d3a190\" (UID: \"f76a9936-58a5-4203-a1c8-9d8d51d3a190\") "
Oct 31 03:57:39.984203 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:39.984019    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f76a9936-58a5-4203-a1c8-9d8d51d3a190-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f76a9936-58a5-4203-a1c8-9d8d51d3a190-maas-api-openshift-ca\") pod \"f76a9936-58a5-4203-a1c8-9d8d51d3a190\" (UID: \"f76a9936-58a5-4203-a1c8-9d8d51d3a190\") "
Oct 31 03:57:39.984203 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:39.984117    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f76a9936-58a5-4203-a1c8-9d8d51d3a190-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f76a9936-58a5-4203-a1c8-9d8d51d3a190-maas-api-xks-ca\") pod \"f76a9936-58a5-4203-a1c8-9d8d51d3a190\" (UID: \"f76a9936-58a5-4203-a1c8-9d8d51d3a190\") "
Oct 31 03:57:39.984375 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:39.984344    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76a9936-58a5-4203-a1c8-9d8d51d3a190-maas-api-openshift-ca" pod "f76a9936-58a5-4203-a1c8-9d8d51d3a190" (UID: "f76a9936-58a5-4203-a1c8-9d8d51d3a190"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:57:39.984648 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:39.984625    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f76a9936-58a5-4203-a1c8-9d8d51d3a190-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:39.986452 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:39.986429    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76a9936-58a5-4203-a1c8-9d8d51d3a190-kube-api-access-nfx7r" pod "f76a9936-58a5-4203-a1c8-9d8d51d3a190" (UID: "f76a9936-58a5-4203-a1c8-9d8d51d3a190"). InnerVolumeSpecName "kube-api-access-nfx7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:57:39.986684 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:39.986662    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f76a9936-58a5-4203-a1c8-9d8d51d3a190-maas-api-xks-ca" pod "f76a9936-58a5-4203-a1c8-9d8d51d3a190" (UID: "f76a9936-58a5-4203-a1c8-9d8d51d3a190"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:57:40.086426 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:40.086389    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f76a9936-58a5-4203-a1c8-9d8d51d3a190-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:40.086426 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:40.086419    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfx7r\" (UniqueName: \"kubernetes.io/projected/f76a9936-58a5-4203-a1c8-9d8d51d3a190-kube-api-access-nfx7r\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:40.725449 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:40.725416    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h"
Oct 31 03:57:40.725449 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:40.725427    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h" event={"ID":"f76a9936-58a5-4203-a1c8-9d8d51d3a190","Type":"ContainerDied","Data":"ee46e9c17e03cebe244d734b742251d48e20e0355a72a83b0ef90e87fc4dfa27"}
Oct 31 03:57:40.725652 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:40.725468    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee46e9c17e03cebe244d734b742251d48e20e0355a72a83b0ef90e87fc4dfa27"
Oct 31 03:57:41.258037 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:41.257999    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"]
Oct 31 03:57:41.258532 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:41.258246    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd" podUID="24d4fdea-5469-4804-801f-3068de198f4a" containerName="istio-proxy" containerID="cri-o://40bf1117e5e680a6b4766224d04ae4f90c592b41ff3bfa789009826cab4be49a" gracePeriod=30
Oct 31 03:57:46.122912 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.120812    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-h526p"]
Oct 31 03:57:46.142246 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.142216    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1180d818-h526p"]
Oct 31 03:57:46.164231 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.164197    4447 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:57:46.164393 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.164280    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd" podUID="24d4fdea-5469-4804-801f-3068de198f4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:57:46.547992 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.547964    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:57:46.612080 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.612047    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edacc995-47b9-4ed7-9a68-7aa8fdce1ce0" path="/var/lib/kubelet/pods/edacc995-47b9-4ed7-9a68-7aa8fdce1ce0/volumes"
Oct 31 03:57:46.625593 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.625567    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24d4fdea-5469-4804-801f-3068de198f4a-istio-token\" (UniqueName: \"kubernetes.io/projected/24d4fdea-5469-4804-801f-3068de198f4a-istio-token\") pod \"24d4fdea-5469-4804-801f-3068de198f4a\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") "
Oct 31 03:57:46.625767 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.625616    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24d4fdea-5469-4804-801f-3068de198f4a-kube-api-access-2cdmv\" (UniqueName: \"kubernetes.io/projected/24d4fdea-5469-4804-801f-3068de198f4a-kube-api-access-2cdmv\") pod \"24d4fdea-5469-4804-801f-3068de198f4a\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") "
Oct 31 03:57:46.625767 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.625746    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-credential-socket\") pod \"24d4fdea-5469-4804-801f-3068de198f4a\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") "
Oct 31 03:57:46.625869 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.625775    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-istio-envoy\") pod \"24d4fdea-5469-4804-801f-3068de198f4a\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") "
Oct 31 03:57:46.625869 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.625830    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-istio-data\") pod \"24d4fdea-5469-4804-801f-3068de198f4a\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") "
Oct 31 03:57:46.625994 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.625868    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-workload-certs\") pod \"24d4fdea-5469-4804-801f-3068de198f4a\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") "
Oct 31 03:57:46.625994 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.625956    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24d4fdea-5469-4804-801f-3068de198f4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24d4fdea-5469-4804-801f-3068de198f4a-istiod-ca-cert\") pod \"24d4fdea-5469-4804-801f-3068de198f4a\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") "
Oct 31 03:57:46.625994 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.625986    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-workload-socket\") pod \"24d4fdea-5469-4804-801f-3068de198f4a\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") "
Oct 31 03:57:46.626145 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.626023    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24d4fdea-5469-4804-801f-3068de198f4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24d4fdea-5469-4804-801f-3068de198f4a-istio-podinfo\") pod \"24d4fdea-5469-4804-801f-3068de198f4a\" (UID: \"24d4fdea-5469-4804-801f-3068de198f4a\") "
Oct 31 03:57:46.626145 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.626050    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-credential-socket" pod "24d4fdea-5469-4804-801f-3068de198f4a" (UID: "24d4fdea-5469-4804-801f-3068de198f4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:57:46.626250 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.626240    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-workload-certs" pod "24d4fdea-5469-4804-801f-3068de198f4a" (UID: "24d4fdea-5469-4804-801f-3068de198f4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:57:46.626825 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.626561    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-workload-socket" pod "24d4fdea-5469-4804-801f-3068de198f4a" (UID: "24d4fdea-5469-4804-801f-3068de198f4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:57:46.626825 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.626749    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:46.626825 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.626769    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:46.626825 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.626782    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:46.626825 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.626802    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d4fdea-5469-4804-801f-3068de198f4a-istiod-ca-cert" pod "24d4fdea-5469-4804-801f-3068de198f4a" (UID: "24d4fdea-5469-4804-801f-3068de198f4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:57:46.627209 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.626836    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-istio-data" pod "24d4fdea-5469-4804-801f-3068de198f4a" (UID: "24d4fdea-5469-4804-801f-3068de198f4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:57:46.628792 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.628658    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d4fdea-5469-4804-801f-3068de198f4a-kube-api-access-2cdmv" pod "24d4fdea-5469-4804-801f-3068de198f4a" (UID: "24d4fdea-5469-4804-801f-3068de198f4a"). InnerVolumeSpecName "kube-api-access-2cdmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:57:46.628792 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.628686    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d4fdea-5469-4804-801f-3068de198f4a-istio-token" pod "24d4fdea-5469-4804-801f-3068de198f4a" (UID: "24d4fdea-5469-4804-801f-3068de198f4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:57:46.628792 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.628785    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24d4fdea-5469-4804-801f-3068de198f4a-istio-podinfo" pod "24d4fdea-5469-4804-801f-3068de198f4a" (UID: "24d4fdea-5469-4804-801f-3068de198f4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:57:46.629144 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.629122    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-istio-envoy" pod "24d4fdea-5469-4804-801f-3068de198f4a" (UID: "24d4fdea-5469-4804-801f-3068de198f4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:57:46.728386 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.728322    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cdmv\" (UniqueName: \"kubernetes.io/projected/24d4fdea-5469-4804-801f-3068de198f4a-kube-api-access-2cdmv\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:46.728386 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.728362    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:46.728386 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.728376    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24d4fdea-5469-4804-801f-3068de198f4a-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:46.728386 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.728389    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24d4fdea-5469-4804-801f-3068de198f4a-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:46.728777 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.728403    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24d4fdea-5469-4804-801f-3068de198f4a-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:46.728777 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.728416    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24d4fdea-5469-4804-801f-3068de198f4a-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:46.792327 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.792296    4447 generic.go:356] "Generic (PLEG): container finished" podID="24d4fdea-5469-4804-801f-3068de198f4a" containerID="40bf1117e5e680a6b4766224d04ae4f90c592b41ff3bfa789009826cab4be49a" exitCode=0
Oct 31 03:57:46.792523 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.792341    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"
Oct 31 03:57:46.792523 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.792370    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd" event={"ID":"24d4fdea-5469-4804-801f-3068de198f4a","Type":"ContainerDied","Data":"40bf1117e5e680a6b4766224d04ae4f90c592b41ff3bfa789009826cab4be49a"}
Oct 31 03:57:46.792523 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.792421    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd" event={"ID":"24d4fdea-5469-4804-801f-3068de198f4a","Type":"ContainerDied","Data":"37abd9b2d9372e965d27573baf37dbd14c215d67508b1812cbece54f568e9558"}
Oct 31 03:57:46.792523 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.792441    4447 scope.go:122] "RemoveContainer" containerID="40bf1117e5e680a6b4766224d04ae4f90c592b41ff3bfa789009826cab4be49a"
Oct 31 03:57:46.804573 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.804553    4447 scope.go:122] "RemoveContainer" containerID="40bf1117e5e680a6b4766224d04ae4f90c592b41ff3bfa789009826cab4be49a"
Oct 31 03:57:46.804864 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:57:46.804845    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40bf1117e5e680a6b4766224d04ae4f90c592b41ff3bfa789009826cab4be49a\": container with ID starting with 40bf1117e5e680a6b4766224d04ae4f90c592b41ff3bfa789009826cab4be49a not found: ID does not exist" containerID="40bf1117e5e680a6b4766224d04ae4f90c592b41ff3bfa789009826cab4be49a"
Oct 31 03:57:46.804979 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.804871    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40bf1117e5e680a6b4766224d04ae4f90c592b41ff3bfa789009826cab4be49a"} err="failed to get container status \"40bf1117e5e680a6b4766224d04ae4f90c592b41ff3bfa789009826cab4be49a\": rpc error: code = NotFound desc = could not find container \"40bf1117e5e680a6b4766224d04ae4f90c592b41ff3bfa789009826cab4be49a\": container with ID starting with 40bf1117e5e680a6b4766224d04ae4f90c592b41ff3bfa789009826cab4be49a not found: ID does not exist"
Oct 31 03:57:46.962944 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.962892    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"]
Oct 31 03:57:46.967501 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:46.967470    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-j5ppd"]
Oct 31 03:57:47.373907 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:47.373794    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84e130-nwtzj"]
Oct 31 03:57:47.378745 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:47.378722    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-84e130-nwtzj"]
Oct 31 03:57:48.049559 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:48.049524    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ad7fb88-tcskb"]
Oct 31 03:57:48.098747 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:48.098717    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ad7fb88-tcskb"]
Oct 31 03:57:48.609566 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:48.609535    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce" path="/var/lib/kubelet/pods/1bd0924a-ecc1-4e38-b279-4a45a6fbc1ce/volumes"
Oct 31 03:57:48.610055 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:48.610040    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24d4fdea-5469-4804-801f-3068de198f4a" path="/var/lib/kubelet/pods/24d4fdea-5469-4804-801f-3068de198f4a/volumes"
Oct 31 03:57:48.610455 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:48.610440    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffe34fa0-f324-4227-a427-bf7228bfbecf" path="/var/lib/kubelet/pods/ffe34fa0-f324-4227-a427-bf7228bfbecf/volumes"
Oct 31 03:57:48.871848 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:48.871760    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb"]
Oct 31 03:57:48.985517 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:48.985484    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx"]
Oct 31 03:57:49.011320 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:57:49.011284    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:57:53.872653 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:53.872594    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb" podUID="76378008-52e0-44da-9bf4-7c6745178755" containerName="payload-processing" containerID="cri-o://96e50bd548219fc976f2ed3c77c618c43f17a9701865cb558e52bc23a4ddd067" gracePeriod=25
Oct 31 03:57:53.986847 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:53.986793    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx" podUID="f334da75-10b1-4320-9d31-4b8d428040d6" containerName="payload-pre-processing" containerID="cri-o://893b776234ab7089837dcfe4e1fd130afee3bc55f6776dc9a02e7d422bd9fb19" gracePeriod=25
Oct 31 03:57:54.161059 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.161024    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb"
Oct 31 03:57:54.237176 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.237137    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76378008-52e0-44da-9bf4-7c6745178755-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76378008-52e0-44da-9bf4-7c6745178755-plugins-config-volume\") pod \"76378008-52e0-44da-9bf4-7c6745178755\" (UID: \"76378008-52e0-44da-9bf4-7c6745178755\") "
Oct 31 03:57:54.237375 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.237259    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76378008-52e0-44da-9bf4-7c6745178755-kube-api-access-qrgc8\" (UniqueName: \"kubernetes.io/projected/76378008-52e0-44da-9bf4-7c6745178755-kube-api-access-qrgc8\") pod \"76378008-52e0-44da-9bf4-7c6745178755\" (UID: \"76378008-52e0-44da-9bf4-7c6745178755\") "
Oct 31 03:57:54.237538 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.237513    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76378008-52e0-44da-9bf4-7c6745178755-plugins-config-volume" pod "76378008-52e0-44da-9bf4-7c6745178755" (UID: "76378008-52e0-44da-9bf4-7c6745178755"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:57:54.237955 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.237930    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76378008-52e0-44da-9bf4-7c6745178755-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:54.239717 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.239693    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76378008-52e0-44da-9bf4-7c6745178755-kube-api-access-qrgc8" pod "76378008-52e0-44da-9bf4-7c6745178755" (UID: "76378008-52e0-44da-9bf4-7c6745178755"). InnerVolumeSpecName "kube-api-access-qrgc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:57:54.259582 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.259561    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx"
Oct 31 03:57:54.338640 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.338608    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f334da75-10b1-4320-9d31-4b8d428040d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f334da75-10b1-4320-9d31-4b8d428040d6-plugins-config-volume\") pod \"f334da75-10b1-4320-9d31-4b8d428040d6\" (UID: \"f334da75-10b1-4320-9d31-4b8d428040d6\") "
Oct 31 03:57:54.338829 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.338712    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f334da75-10b1-4320-9d31-4b8d428040d6-kube-api-access-qh9hz\" (UniqueName: \"kubernetes.io/projected/f334da75-10b1-4320-9d31-4b8d428040d6-kube-api-access-qh9hz\") pod \"f334da75-10b1-4320-9d31-4b8d428040d6\" (UID: \"f334da75-10b1-4320-9d31-4b8d428040d6\") "
Oct 31 03:57:54.339064 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.339042    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f334da75-10b1-4320-9d31-4b8d428040d6-plugins-config-volume" pod "f334da75-10b1-4320-9d31-4b8d428040d6" (UID: "f334da75-10b1-4320-9d31-4b8d428040d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:57:54.339352 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.339335    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrgc8\" (UniqueName: \"kubernetes.io/projected/76378008-52e0-44da-9bf4-7c6745178755-kube-api-access-qrgc8\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:54.339416 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.339357    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f334da75-10b1-4320-9d31-4b8d428040d6-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:54.341204 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.341185    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f334da75-10b1-4320-9d31-4b8d428040d6-kube-api-access-qh9hz" pod "f334da75-10b1-4320-9d31-4b8d428040d6" (UID: "f334da75-10b1-4320-9d31-4b8d428040d6"). InnerVolumeSpecName "kube-api-access-qh9hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:57:54.441303 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.441225    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh9hz\" (UniqueName: \"kubernetes.io/projected/f334da75-10b1-4320-9d31-4b8d428040d6-kube-api-access-qh9hz\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:57:54.855281 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.855243    4447 generic.go:356] "Generic (PLEG): container finished" podID="76378008-52e0-44da-9bf4-7c6745178755" containerID="96e50bd548219fc976f2ed3c77c618c43f17a9701865cb558e52bc23a4ddd067" exitCode=0
Oct 31 03:57:54.855458 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.855296    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb"
Oct 31 03:57:54.855458 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.855337    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb" event={"ID":"76378008-52e0-44da-9bf4-7c6745178755","Type":"ContainerDied","Data":"96e50bd548219fc976f2ed3c77c618c43f17a9701865cb558e52bc23a4ddd067"}
Oct 31 03:57:54.855458 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.855395    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb" event={"ID":"76378008-52e0-44da-9bf4-7c6745178755","Type":"ContainerDied","Data":"b8d44e98fbaeb72fb89bdbf8e48cd68377d7288ecb9a5e41b2f799b6febb3a21"}
Oct 31 03:57:54.855458 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.855419    4447 scope.go:122] "RemoveContainer" containerID="96e50bd548219fc976f2ed3c77c618c43f17a9701865cb558e52bc23a4ddd067"
Oct 31 03:57:54.857165 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.857139    4447 generic.go:356] "Generic (PLEG): container finished" podID="f334da75-10b1-4320-9d31-4b8d428040d6" containerID="893b776234ab7089837dcfe4e1fd130afee3bc55f6776dc9a02e7d422bd9fb19" exitCode=0
Oct 31 03:57:54.857301 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.857175    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx" event={"ID":"f334da75-10b1-4320-9d31-4b8d428040d6","Type":"ContainerDied","Data":"893b776234ab7089837dcfe4e1fd130afee3bc55f6776dc9a02e7d422bd9fb19"}
Oct 31 03:57:54.857301 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.857184    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx"
Oct 31 03:57:54.857301 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.857211    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx" event={"ID":"f334da75-10b1-4320-9d31-4b8d428040d6","Type":"ContainerDied","Data":"4ed6d2c566b23a6e33cdd39f89d0c4f62a8a745bce4e4eb81e95e35c20cda8dd"}
Oct 31 03:57:54.866547 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.866530    4447 scope.go:122] "RemoveContainer" containerID="96e50bd548219fc976f2ed3c77c618c43f17a9701865cb558e52bc23a4ddd067"
Oct 31 03:57:54.866850 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:57:54.866831    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96e50bd548219fc976f2ed3c77c618c43f17a9701865cb558e52bc23a4ddd067\": container with ID starting with 96e50bd548219fc976f2ed3c77c618c43f17a9701865cb558e52bc23a4ddd067 not found: ID does not exist" containerID="96e50bd548219fc976f2ed3c77c618c43f17a9701865cb558e52bc23a4ddd067"
Oct 31 03:57:54.866934 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.866858    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e50bd548219fc976f2ed3c77c618c43f17a9701865cb558e52bc23a4ddd067"} err="failed to get container status \"96e50bd548219fc976f2ed3c77c618c43f17a9701865cb558e52bc23a4ddd067\": rpc error: code = NotFound desc = could not find container \"96e50bd548219fc976f2ed3c77c618c43f17a9701865cb558e52bc23a4ddd067\": container with ID starting with 96e50bd548219fc976f2ed3c77c618c43f17a9701865cb558e52bc23a4ddd067 not found: ID does not exist"
Oct 31 03:57:54.866934 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.866876    4447 scope.go:122] "RemoveContainer" containerID="893b776234ab7089837dcfe4e1fd130afee3bc55f6776dc9a02e7d422bd9fb19"
Oct 31 03:57:54.876437 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.876301    4447 scope.go:122] "RemoveContainer" containerID="893b776234ab7089837dcfe4e1fd130afee3bc55f6776dc9a02e7d422bd9fb19"
Oct 31 03:57:54.876712 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:57:54.876540    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"893b776234ab7089837dcfe4e1fd130afee3bc55f6776dc9a02e7d422bd9fb19\": container with ID starting with 893b776234ab7089837dcfe4e1fd130afee3bc55f6776dc9a02e7d422bd9fb19 not found: ID does not exist" containerID="893b776234ab7089837dcfe4e1fd130afee3bc55f6776dc9a02e7d422bd9fb19"
Oct 31 03:57:54.876712 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:54.876572    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"893b776234ab7089837dcfe4e1fd130afee3bc55f6776dc9a02e7d422bd9fb19"} err="failed to get container status \"893b776234ab7089837dcfe4e1fd130afee3bc55f6776dc9a02e7d422bd9fb19\": rpc error: code = NotFound desc = could not find container \"893b776234ab7089837dcfe4e1fd130afee3bc55f6776dc9a02e7d422bd9fb19\": container with ID starting with 893b776234ab7089837dcfe4e1fd130afee3bc55f6776dc9a02e7d422bd9fb19 not found: ID does not exist"
Oct 31 03:57:55.022791 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:55.022761    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb"]
Oct 31 03:57:55.073323 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:55.073295    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-1783b3-785658497f-djrsb"]
Oct 31 03:57:55.139070 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:55.138981    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx"]
Oct 31 03:57:55.144470 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:55.144441    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1783b3-8955df67f-w4nrx"]
Oct 31 03:57:56.615469 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:56.615435    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76378008-52e0-44da-9bf4-7c6745178755" path="/var/lib/kubelet/pods/76378008-52e0-44da-9bf4-7c6745178755/volumes"
Oct 31 03:57:56.615932 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:57:56.615909    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f334da75-10b1-4320-9d31-4b8d428040d6" path="/var/lib/kubelet/pods/f334da75-10b1-4320-9d31-4b8d428040d6/volumes"
Oct 31 03:58:06.994833 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:06.994799    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn"]
Oct 31 03:58:07.072161 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:07.072128    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq"]
Oct 31 03:58:11.995498 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:11.995453    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn" podUID="683a25bd-a64b-4a4c-8dbc-df6a1dd431aa" containerName="payload-processing" containerID="cri-o://49c4dd056266b8b036bb0755ca7dc68d88fadffcf392bc151e27ebea4e268439" gracePeriod=25
Oct 31 03:58:12.073108 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:12.073069    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq" podUID="f3b51253-b4d7-49c9-be26-36a4e5320c82" containerName="payload-pre-processing" containerID="cri-o://d91a31cbe8344153057de52fbc4cdd2b4d7c38669fbe3e232991ee0556734325" gracePeriod=25
Oct 31 03:58:12.347344 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:12.347312    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn"
Oct 31 03:58:12.382397 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:12.382375    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq"
Oct 31 03:58:12.447816 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:12.447781    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/683a25bd-a64b-4a4c-8dbc-df6a1dd431aa-kube-api-access-wwgqk\" (UniqueName: \"kubernetes.io/projected/683a25bd-a64b-4a4c-8dbc-df6a1dd431aa-kube-api-access-wwgqk\") pod \"683a25bd-a64b-4a4c-8dbc-df6a1dd431aa\" (UID: \"683a25bd-a64b-4a4c-8dbc-df6a1dd431aa\") "
Oct 31 03:58:12.448019 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:12.447867    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b51253-b4d7-49c9-be26-36a4e5320c82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b51253-b4d7-49c9-be26-36a4e5320c82-plugins-config-volume\") pod \"f3b51253-b4d7-49c9-be26-36a4e5320c82\" (UID: \"f3b51253-b4d7-49c9-be26-36a4e5320c82\") "
Oct 31 03:58:12.448019 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:12.447981    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/683a25bd-a64b-4a4c-8dbc-df6a1dd431aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/683a25bd-a64b-4a4c-8dbc-df6a1dd431aa-plugins-config-volume\") pod \"683a25bd-a64b-4a4c-8dbc-df6a1dd431aa\" (UID: \"683a25bd-a64b-4a4c-8dbc-df6a1dd431aa\") "
Oct 31 03:58:12.448116 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:12.448091    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b51253-b4d7-49c9-be26-36a4e5320c82-kube-api-access-82prv\" (UniqueName: \"kubernetes.io/projected/f3b51253-b4d7-49c9-be26-36a4e5320c82-kube-api-access-82prv\") pod \"f3b51253-b4d7-49c9-be26-36a4e5320c82\" (UID: \"f3b51253-b4d7-49c9-be26-36a4e5320c82\") "
Oct 31 03:58:12.448307 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:12.448275    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b51253-b4d7-49c9-be26-36a4e5320c82-plugins-config-volume" pod "f3b51253-b4d7-49c9-be26-36a4e5320c82" (UID: "f3b51253-b4d7-49c9-be26-36a4e5320c82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:58:12.448451 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:12.448330    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/683a25bd-a64b-4a4c-8dbc-df6a1dd431aa-plugins-config-volume" pod "683a25bd-a64b-4a4c-8dbc-df6a1dd431aa" (UID: "683a25bd-a64b-4a4c-8dbc-df6a1dd431aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:58:12.448623 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:12.448604    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3b51253-b4d7-49c9-be26-36a4e5320c82-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:12.448689 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:12.448629    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/683a25bd-a64b-4a4c-8dbc-df6a1dd431aa-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:12.450284 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:12.450259    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/683a25bd-a64b-4a4c-8dbc-df6a1dd431aa-kube-api-access-wwgqk" pod "683a25bd-a64b-4a4c-8dbc-df6a1dd431aa" (UID: "683a25bd-a64b-4a4c-8dbc-df6a1dd431aa"). InnerVolumeSpecName "kube-api-access-wwgqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:58:12.450399 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:12.450364    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b51253-b4d7-49c9-be26-36a4e5320c82-kube-api-access-82prv" pod "f3b51253-b4d7-49c9-be26-36a4e5320c82" (UID: "f3b51253-b4d7-49c9-be26-36a4e5320c82"). InnerVolumeSpecName "kube-api-access-82prv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:58:12.549744 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:12.549653    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82prv\" (UniqueName: \"kubernetes.io/projected/f3b51253-b4d7-49c9-be26-36a4e5320c82-kube-api-access-82prv\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:12.549744 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:12.549685    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwgqk\" (UniqueName: \"kubernetes.io/projected/683a25bd-a64b-4a4c-8dbc-df6a1dd431aa-kube-api-access-wwgqk\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:13.036763 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:13.036730    4447 generic.go:356] "Generic (PLEG): container finished" podID="f3b51253-b4d7-49c9-be26-36a4e5320c82" containerID="d91a31cbe8344153057de52fbc4cdd2b4d7c38669fbe3e232991ee0556734325" exitCode=0
Oct 31 03:58:13.037221 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:13.036772    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq"
Oct 31 03:58:13.037221 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:13.036821    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq" event={"ID":"f3b51253-b4d7-49c9-be26-36a4e5320c82","Type":"ContainerDied","Data":"d91a31cbe8344153057de52fbc4cdd2b4d7c38669fbe3e232991ee0556734325"}
Oct 31 03:58:13.037221 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:13.036873    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq" event={"ID":"f3b51253-b4d7-49c9-be26-36a4e5320c82","Type":"ContainerDied","Data":"0fc31e09ecb7ea37250288ffd048c6dbbfba93ab5ecae81860069bf3e3ba21a8"}
Oct 31 03:58:13.037221 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:13.036918    4447 scope.go:122] "RemoveContainer" containerID="d91a31cbe8344153057de52fbc4cdd2b4d7c38669fbe3e232991ee0556734325"
Oct 31 03:58:13.038791 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:13.038764    4447 generic.go:356] "Generic (PLEG): container finished" podID="683a25bd-a64b-4a4c-8dbc-df6a1dd431aa" containerID="49c4dd056266b8b036bb0755ca7dc68d88fadffcf392bc151e27ebea4e268439" exitCode=0
Oct 31 03:58:13.039002 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:13.038820    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn"
Oct 31 03:58:13.039002 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:13.038868    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn" event={"ID":"683a25bd-a64b-4a4c-8dbc-df6a1dd431aa","Type":"ContainerDied","Data":"49c4dd056266b8b036bb0755ca7dc68d88fadffcf392bc151e27ebea4e268439"}
Oct 31 03:58:13.039148 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:13.039006    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn" event={"ID":"683a25bd-a64b-4a4c-8dbc-df6a1dd431aa","Type":"ContainerDied","Data":"893a3229e4d1c7f77eba7f81f03a1d51b08a86d7ca1996d8a69fbbd45e00a272"}
Oct 31 03:58:13.047614 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:13.047597    4447 scope.go:122] "RemoveContainer" containerID="d91a31cbe8344153057de52fbc4cdd2b4d7c38669fbe3e232991ee0556734325"
Oct 31 03:58:13.047875 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:58:13.047859    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d91a31cbe8344153057de52fbc4cdd2b4d7c38669fbe3e232991ee0556734325\": container with ID starting with d91a31cbe8344153057de52fbc4cdd2b4d7c38669fbe3e232991ee0556734325 not found: ID does not exist" containerID="d91a31cbe8344153057de52fbc4cdd2b4d7c38669fbe3e232991ee0556734325"
Oct 31 03:58:13.047966 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:13.047901    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d91a31cbe8344153057de52fbc4cdd2b4d7c38669fbe3e232991ee0556734325"} err="failed to get container status \"d91a31cbe8344153057de52fbc4cdd2b4d7c38669fbe3e232991ee0556734325\": rpc error: code = NotFound desc = could not find container \"d91a31cbe8344153057de52fbc4cdd2b4d7c38669fbe3e232991ee0556734325\": container with ID starting with d91a31cbe8344153057de52fbc4cdd2b4d7c38669fbe3e232991ee0556734325 not found: ID does not exist"
Oct 31 03:58:13.047966 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:13.047929    4447 scope.go:122] "RemoveContainer" containerID="49c4dd056266b8b036bb0755ca7dc68d88fadffcf392bc151e27ebea4e268439"
Oct 31 03:58:13.057418 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:13.057401    4447 scope.go:122] "RemoveContainer" containerID="49c4dd056266b8b036bb0755ca7dc68d88fadffcf392bc151e27ebea4e268439"
Oct 31 03:58:13.057651 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:58:13.057628    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49c4dd056266b8b036bb0755ca7dc68d88fadffcf392bc151e27ebea4e268439\": container with ID starting with 49c4dd056266b8b036bb0755ca7dc68d88fadffcf392bc151e27ebea4e268439 not found: ID does not exist" containerID="49c4dd056266b8b036bb0755ca7dc68d88fadffcf392bc151e27ebea4e268439"
Oct 31 03:58:13.057741 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:13.057656    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49c4dd056266b8b036bb0755ca7dc68d88fadffcf392bc151e27ebea4e268439"} err="failed to get container status \"49c4dd056266b8b036bb0755ca7dc68d88fadffcf392bc151e27ebea4e268439\": rpc error: code = NotFound desc = could not find container \"49c4dd056266b8b036bb0755ca7dc68d88fadffcf392bc151e27ebea4e268439\": container with ID starting with 49c4dd056266b8b036bb0755ca7dc68d88fadffcf392bc151e27ebea4e268439 not found: ID does not exist"
Oct 31 03:58:13.211731 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:13.211695    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq"]
Oct 31 03:58:13.262419 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:13.262329    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b16de9-5f49c9dbd7-7m7cq"]
Oct 31 03:58:13.337109 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:13.337074    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn"]
Oct 31 03:58:13.342594 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:13.342567    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b16de9-5b8f8dd9cf-mfhjn"]
Oct 31 03:58:14.610593 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:14.610560    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="683a25bd-a64b-4a4c-8dbc-df6a1dd431aa" path="/var/lib/kubelet/pods/683a25bd-a64b-4a4c-8dbc-df6a1dd431aa/volumes"
Oct 31 03:58:14.611116 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:14.611100    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b51253-b4d7-49c9-be26-36a4e5320c82" path="/var/lib/kubelet/pods/f3b51253-b4d7-49c9-be26-36a4e5320c82/volumes"
Oct 31 03:58:19.002897 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:58:19.002842    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:58:21.050613 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:21.050557    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba367b2-kkk7d"]
Oct 31 03:58:21.101592 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:21.101559    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d114f70b-lcntl"]
Oct 31 03:58:21.106282 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:21.106253    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba367b2-kkk7d"]
Oct 31 03:58:21.109796 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:21.109772    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d114f70b-lcntl"]
Oct 31 03:58:22.610942 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:22.610899    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cb4f9a2-dfba-43a6-8d42-a68281d62d37" path="/var/lib/kubelet/pods/0cb4f9a2-dfba-43a6-8d42-a68281d62d37/volumes"
Oct 31 03:58:22.611533 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:22.611511    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ea92bdf-3a68-49be-80fe-b32ea7e220a2" path="/var/lib/kubelet/pods/0ea92bdf-3a68-49be-80fe-b32ea7e220a2/volumes"
Oct 31 03:58:28.768467 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:28.768428    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h"]
Oct 31 03:58:28.818526 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:28.818489    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1783b3-gk56h"]
Oct 31 03:58:30.609685 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:30.609647    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f76a9936-58a5-4203-a1c8-9d8d51d3a190" path="/var/lib/kubelet/pods/f76a9936-58a5-4203-a1c8-9d8d51d3a190/volumes"
Oct 31 03:58:35.606899 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:35.606848    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp"]
Oct 31 03:58:35.656024 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:35.655990    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e02dd5-qmddp"]
Oct 31 03:58:35.890468 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:35.890390    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"]
Oct 31 03:58:35.890616 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:35.890593    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh" podUID="dd66aaca-cbe4-414b-a056-551ed0c3f0b6" containerName="istio-proxy" containerID="cri-o://e0d9f58e6e6191486d5684e921dbac4c322090bd35d5470b1ee796ed4d033b59" gracePeriod=30
Oct 31 03:58:36.616925 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:36.616866    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c8ae678-217a-4501-8cd1-646f7a169f3e" path="/var/lib/kubelet/pods/3c8ae678-217a-4501-8cd1-646f7a169f3e/volumes"
Oct 31 03:58:36.633371 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:36.633332    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"]
Oct 31 03:58:36.638256 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:36.638210    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj" podUID="27ca7445-d121-4ae9-abe7-bbf2643ab453" containerName="istio-proxy" containerID="cri-o://9f0877ad7fe748db3532567b154c0c3b1cef30d3af87dcc3f33b034fc25175be" gracePeriod=30
Oct 31 03:58:36.640233 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:36.640206    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"]
Oct 31 03:58:36.640476 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:36.640423    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh" podUID="a02c6569-f6f4-41ac-bcb1-cd42873c13b2" containerName="istio-proxy" containerID="cri-o://7e483ad7884cbc2c841b46bb02a0dfbde8f6200d94708c0c38282f616d74d5b9" gracePeriod=30
Oct 31 03:58:36.670402 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:36.670372    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"]
Oct 31 03:58:36.670602 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:36.670569    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s" podUID="e3f1c261-eea4-45ad-ad44-b0f997723e87" containerName="istio-proxy" containerID="cri-o://097886db4be0667859395298df95e8ab6865a2a7aff2e8bab0da330650ae5686" gracePeriod=30
Oct 31 03:58:36.674560 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:36.674534    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"]
Oct 31 03:58:36.674741 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:36.674721    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m" podUID="cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f" containerName="istio-proxy" containerID="cri-o://b250474413cfbbf3a78cd51a679c7816522ac53451fd6624685caba20cac8f6e" gracePeriod=30
Oct 31 03:58:36.689898 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:36.689849    4447 scope.go:122] "RemoveContainer" containerID="c15e5507bbbe11a8601252b30f00e8e81777356cc8d8102b57224602a8fff97a"
Oct 31 03:58:36.700930 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:36.700861    4447 scope.go:122] "RemoveContainer" containerID="723ca56d5cc404233a2b1865b6b758d6ac2a39c65696acaa27bfae766aabe009"
Oct 31 03:58:41.178492 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.178461    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:58:41.234220 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.234187    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-credential-socket\") pod \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") "
Oct 31 03:58:41.234408 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.234298    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-token\" (UniqueName: \"kubernetes.io/projected/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-token\") pod \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") "
Oct 31 03:58:41.234408 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.234330    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-envoy\") pod \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") "
Oct 31 03:58:41.234408 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.234348    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-workload-socket\") pod \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") "
Oct 31 03:58:41.234408 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.234381    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-workload-certs\") pod \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") "
Oct 31 03:58:41.234635 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.234424    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-data\") pod \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") "
Oct 31 03:58:41.234635 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.234455    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istiod-ca-cert\") pod \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") "
Oct 31 03:58:41.234635 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.234482    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-podinfo\") pod \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") "
Oct 31 03:58:41.234635 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.234504    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-credential-socket" pod "dd66aaca-cbe4-414b-a056-551ed0c3f0b6" (UID: "dd66aaca-cbe4-414b-a056-551ed0c3f0b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:41.234635 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.234579    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-kube-api-access-n5g8q\" (UniqueName: \"kubernetes.io/projected/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-kube-api-access-n5g8q\") pod \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\" (UID: \"dd66aaca-cbe4-414b-a056-551ed0c3f0b6\") "
Oct 31 03:58:41.234866 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.234694    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-workload-socket" pod "dd66aaca-cbe4-414b-a056-551ed0c3f0b6" (UID: "dd66aaca-cbe4-414b-a056-551ed0c3f0b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:41.234866 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.234715    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-workload-certs" pod "dd66aaca-cbe4-414b-a056-551ed0c3f0b6" (UID: "dd66aaca-cbe4-414b-a056-551ed0c3f0b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:41.235082 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.235056    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istiod-ca-cert" pod "dd66aaca-cbe4-414b-a056-551ed0c3f0b6" (UID: "dd66aaca-cbe4-414b-a056-551ed0c3f0b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:58:41.235567 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.235543    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:41.235689 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.235571    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:41.235689 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.235587    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:41.235689 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.235600    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:41.235689 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.235673    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-data" pod "dd66aaca-cbe4-414b-a056-551ed0c3f0b6" (UID: "dd66aaca-cbe4-414b-a056-551ed0c3f0b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:41.237014 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.236990    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-envoy" pod "dd66aaca-cbe4-414b-a056-551ed0c3f0b6" (UID: "dd66aaca-cbe4-414b-a056-551ed0c3f0b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:41.237101 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.237018    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-kube-api-access-n5g8q" pod "dd66aaca-cbe4-414b-a056-551ed0c3f0b6" (UID: "dd66aaca-cbe4-414b-a056-551ed0c3f0b6"). InnerVolumeSpecName "kube-api-access-n5g8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:58:41.237101 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.237041    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-token" pod "dd66aaca-cbe4-414b-a056-551ed0c3f0b6" (UID: "dd66aaca-cbe4-414b-a056-551ed0c3f0b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:58:41.237101 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.237054    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-podinfo" pod "dd66aaca-cbe4-414b-a056-551ed0c3f0b6" (UID: "dd66aaca-cbe4-414b-a056-551ed0c3f0b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:58:41.285321 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.285284    4447 generic.go:356] "Generic (PLEG): container finished" podID="dd66aaca-cbe4-414b-a056-551ed0c3f0b6" containerID="e0d9f58e6e6191486d5684e921dbac4c322090bd35d5470b1ee796ed4d033b59" exitCode=0
Oct 31 03:58:41.285504 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.285338    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"
Oct 31 03:58:41.285504 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.285373    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh" event={"ID":"dd66aaca-cbe4-414b-a056-551ed0c3f0b6","Type":"ContainerDied","Data":"e0d9f58e6e6191486d5684e921dbac4c322090bd35d5470b1ee796ed4d033b59"}
Oct 31 03:58:41.285504 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.285435    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh" event={"ID":"dd66aaca-cbe4-414b-a056-551ed0c3f0b6","Type":"ContainerDied","Data":"bb1ef9c9ecbb3dfc298a0aa31bf7345beddbd6fdb7ec0631e405edf2a91aaafe"}
Oct 31 03:58:41.285504 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.285463    4447 scope.go:122] "RemoveContainer" containerID="e0d9f58e6e6191486d5684e921dbac4c322090bd35d5470b1ee796ed4d033b59"
Oct 31 03:58:41.298502 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.298480    4447 scope.go:122] "RemoveContainer" containerID="e0d9f58e6e6191486d5684e921dbac4c322090bd35d5470b1ee796ed4d033b59"
Oct 31 03:58:41.298810 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:58:41.298790    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0d9f58e6e6191486d5684e921dbac4c322090bd35d5470b1ee796ed4d033b59\": container with ID starting with e0d9f58e6e6191486d5684e921dbac4c322090bd35d5470b1ee796ed4d033b59 not found: ID does not exist" containerID="e0d9f58e6e6191486d5684e921dbac4c322090bd35d5470b1ee796ed4d033b59"
Oct 31 03:58:41.298866 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.298821    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0d9f58e6e6191486d5684e921dbac4c322090bd35d5470b1ee796ed4d033b59"} err="failed to get container status \"e0d9f58e6e6191486d5684e921dbac4c322090bd35d5470b1ee796ed4d033b59\": rpc error: code = NotFound desc = could not find container \"e0d9f58e6e6191486d5684e921dbac4c322090bd35d5470b1ee796ed4d033b59\": container with ID starting with e0d9f58e6e6191486d5684e921dbac4c322090bd35d5470b1ee796ed4d033b59 not found: ID does not exist"
Oct 31 03:58:41.301564 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.301542    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"]
Oct 31 03:58:41.336829 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.336794    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:41.336829 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.336821    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:41.336829 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.336837    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:41.337083 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.336846    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:41.337083 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.336854    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5g8q\" (UniqueName: \"kubernetes.io/projected/dd66aaca-cbe4-414b-a056-551ed0c3f0b6-kube-api-access-n5g8q\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:41.467818 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.467780    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"]
Oct 31 03:58:41.474386 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.474360    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1783b3-openshift-default-8b6688df-54pjh"]
Oct 31 03:58:41.972949 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:41.972916    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:58:42.018247 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.018187    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:58:42.045102 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.045078    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:58:42.045395 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.045365    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-podinfo\") pod \"27ca7445-d121-4ae9-abe7-bbf2643ab453\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") "
Oct 31 03:58:42.045487 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.045404    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-workload-certs\") pod \"27ca7445-d121-4ae9-abe7-bbf2643ab453\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") "
Oct 31 03:58:42.045487 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.045438    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ca7445-d121-4ae9-abe7-bbf2643ab453-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ca7445-d121-4ae9-abe7-bbf2643ab453-istiod-ca-cert\") pod \"27ca7445-d121-4ae9-abe7-bbf2643ab453\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") "
Oct 31 03:58:42.045487 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.045473    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-envoy\") pod \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") "
Oct 31 03:58:42.045658 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.045497    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-token\" (UniqueName: \"kubernetes.io/projected/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-token\") pod \"27ca7445-d121-4ae9-abe7-bbf2643ab453\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") "
Oct 31 03:58:42.045658 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.045527    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-workload-socket\") pod \"27ca7445-d121-4ae9-abe7-bbf2643ab453\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") "
Oct 31 03:58:42.045658 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.045573    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-kube-api-access-c96k8\" (UniqueName: \"kubernetes.io/projected/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-kube-api-access-c96k8\") pod \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") "
Oct 31 03:58:42.045658 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.045631    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-credential-socket\") pod \"27ca7445-d121-4ae9-abe7-bbf2643ab453\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") "
Oct 31 03:58:42.045864 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.045669    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-podinfo\") pod \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") "
Oct 31 03:58:42.045864 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.045735    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-workload-certs" pod "27ca7445-d121-4ae9-abe7-bbf2643ab453" (UID: "27ca7445-d121-4ae9-abe7-bbf2643ab453"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:42.045864 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.045763    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-data\") pod \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") "
Oct 31 03:58:42.045864 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.045860    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-envoy\") pod \"27ca7445-d121-4ae9-abe7-bbf2643ab453\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") "
Oct 31 03:58:42.046093 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.045872    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-workload-socket" pod "27ca7445-d121-4ae9-abe7-bbf2643ab453" (UID: "27ca7445-d121-4ae9-abe7-bbf2643ab453"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:42.046093 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.045937    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ca7445-d121-4ae9-abe7-bbf2643ab453-istiod-ca-cert" pod "27ca7445-d121-4ae9-abe7-bbf2643ab453" (UID: "27ca7445-d121-4ae9-abe7-bbf2643ab453"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:58:42.046203 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.045912    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-workload-certs\") pod \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") "
Oct 31 03:58:42.046203 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.046102    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-workload-certs" pod "a02c6569-f6f4-41ac-bcb1-cd42873c13b2" (UID: "a02c6569-f6f4-41ac-bcb1-cd42873c13b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:42.046203 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.046150    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ca7445-d121-4ae9-abe7-bbf2643ab453-kube-api-access-22mr4\" (UniqueName: \"kubernetes.io/projected/27ca7445-d121-4ae9-abe7-bbf2643ab453-kube-api-access-22mr4\") pod \"27ca7445-d121-4ae9-abe7-bbf2643ab453\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") "
Oct 31 03:58:42.046340 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.046243    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-token\" (UniqueName: \"kubernetes.io/projected/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-token\") pod \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") "
Oct 31 03:58:42.046340 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.046297    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istiod-ca-cert\") pod \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") "
Oct 31 03:58:42.046434 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.046347    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-credential-socket\") pod \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") "
Oct 31 03:58:42.046434 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.046380    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-data\") pod \"27ca7445-d121-4ae9-abe7-bbf2643ab453\" (UID: \"27ca7445-d121-4ae9-abe7-bbf2643ab453\") "
Oct 31 03:58:42.046434 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.046413    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-workload-socket\") pod \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\" (UID: \"a02c6569-f6f4-41ac-bcb1-cd42873c13b2\") "
Oct 31 03:58:42.046583 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.046554    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-data" pod "a02c6569-f6f4-41ac-bcb1-cd42873c13b2" (UID: "a02c6569-f6f4-41ac-bcb1-cd42873c13b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:42.047309 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.046840    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-credential-socket" pod "a02c6569-f6f4-41ac-bcb1-cd42873c13b2" (UID: "a02c6569-f6f4-41ac-bcb1-cd42873c13b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:42.047309 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.046918    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istiod-ca-cert" pod "a02c6569-f6f4-41ac-bcb1-cd42873c13b2" (UID: "a02c6569-f6f4-41ac-bcb1-cd42873c13b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:58:42.047309 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.047055    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-workload-socket" pod "a02c6569-f6f4-41ac-bcb1-cd42873c13b2" (UID: "a02c6569-f6f4-41ac-bcb1-cd42873c13b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:42.047309 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.047139    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-data" pod "27ca7445-d121-4ae9-abe7-bbf2643ab453" (UID: "27ca7445-d121-4ae9-abe7-bbf2643ab453"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:42.047309 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.047217    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.047309 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.047237    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.047309 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.047251    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.047309 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.047265    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.047309 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.047279    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.047309 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.047216    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-credential-socket" pod "27ca7445-d121-4ae9-abe7-bbf2643ab453" (UID: "27ca7445-d121-4ae9-abe7-bbf2643ab453"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:42.047309 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.047293    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.048126 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.047341    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ca7445-d121-4ae9-abe7-bbf2643ab453-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.048126 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.047360    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.048419 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.048400    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-podinfo" pod "27ca7445-d121-4ae9-abe7-bbf2643ab453" (UID: "27ca7445-d121-4ae9-abe7-bbf2643ab453"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:58:42.049248 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.049226    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-podinfo" pod "a02c6569-f6f4-41ac-bcb1-cd42873c13b2" (UID: "a02c6569-f6f4-41ac-bcb1-cd42873c13b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:58:42.049343 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.049248    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-kube-api-access-c96k8" pod "a02c6569-f6f4-41ac-bcb1-cd42873c13b2" (UID: "a02c6569-f6f4-41ac-bcb1-cd42873c13b2"). InnerVolumeSpecName "kube-api-access-c96k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:58:42.049343 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.049283    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-token" pod "27ca7445-d121-4ae9-abe7-bbf2643ab453" (UID: "27ca7445-d121-4ae9-abe7-bbf2643ab453"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:58:42.049418 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.049388    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-envoy" pod "27ca7445-d121-4ae9-abe7-bbf2643ab453" (UID: "27ca7445-d121-4ae9-abe7-bbf2643ab453"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:42.049597 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.049575    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-envoy" pod "a02c6569-f6f4-41ac-bcb1-cd42873c13b2" (UID: "a02c6569-f6f4-41ac-bcb1-cd42873c13b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:42.050038 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.050022    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-token" pod "a02c6569-f6f4-41ac-bcb1-cd42873c13b2" (UID: "a02c6569-f6f4-41ac-bcb1-cd42873c13b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:58:42.050567 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.050554    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ca7445-d121-4ae9-abe7-bbf2643ab453-kube-api-access-22mr4" pod "27ca7445-d121-4ae9-abe7-bbf2643ab453" (UID: "27ca7445-d121-4ae9-abe7-bbf2643ab453"). InnerVolumeSpecName "kube-api-access-22mr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:58:42.050830 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.050811    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:58:42.148147 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.148113    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-workload-certs\") pod \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") "
Oct 31 03:58:42.148337 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.148168    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-token\" (UniqueName: \"kubernetes.io/projected/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-token\") pod \"e3f1c261-eea4-45ad-ad44-b0f997723e87\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") "
Oct 31 03:58:42.148337 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.148193    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-data\") pod \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") "
Oct 31 03:58:42.148337 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.148216    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f1c261-eea4-45ad-ad44-b0f997723e87-kube-api-access-xsqjx\" (UniqueName: \"kubernetes.io/projected/e3f1c261-eea4-45ad-ad44-b0f997723e87-kube-api-access-xsqjx\") pod \"e3f1c261-eea4-45ad-ad44-b0f997723e87\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") "
Oct 31 03:58:42.148337 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.148258    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-credential-socket\") pod \"e3f1c261-eea4-45ad-ad44-b0f997723e87\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") "
Oct 31 03:58:42.148662 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.148440    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-workload-certs" pod "cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f" (UID: "cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:42.148662 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.148522    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-podinfo\") pod \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") "
Oct 31 03:58:42.148662 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.148542    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-credential-socket" pod "e3f1c261-eea4-45ad-ad44-b0f997723e87" (UID: "e3f1c261-eea4-45ad-ad44-b0f997723e87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:42.148662 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.148551    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-data" pod "cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f" (UID: "cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:42.148662 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.148618    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-workload-socket\") pod \"e3f1c261-eea4-45ad-ad44-b0f997723e87\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") "
Oct 31 03:58:42.148934 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.148700    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-envoy\") pod \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") "
Oct 31 03:58:42.148934 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.148726    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-credential-socket\") pod \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") "
Oct 31 03:58:42.148934 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.148794    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-data\") pod \"e3f1c261-eea4-45ad-ad44-b0f997723e87\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") "
Oct 31 03:58:42.148934 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.148826    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-token\" (UniqueName: \"kubernetes.io/projected/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-token\") pod \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") "
Oct 31 03:58:42.148934 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.148812    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-workload-socket" pod "e3f1c261-eea4-45ad-ad44-b0f997723e87" (UID: "e3f1c261-eea4-45ad-ad44-b0f997723e87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:42.148934 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.148876    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-workload-certs\") pod \"e3f1c261-eea4-45ad-ad44-b0f997723e87\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") "
Oct 31 03:58:42.149232 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.148973    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istiod-ca-cert\") pod \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") "
Oct 31 03:58:42.149232 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.149011    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3f1c261-eea4-45ad-ad44-b0f997723e87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f1c261-eea4-45ad-ad44-b0f997723e87-istiod-ca-cert\") pod \"e3f1c261-eea4-45ad-ad44-b0f997723e87\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") "
Oct 31 03:58:42.149232 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.149030    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-credential-socket" pod "cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f" (UID: "cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:42.149232 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.149084    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-podinfo\") pod \"e3f1c261-eea4-45ad-ad44-b0f997723e87\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") "
Oct 31 03:58:42.149232 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.149173    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-workload-socket\") pod \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") "
Oct 31 03:58:42.149232 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.149205    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-kube-api-access-6k78g\" (UniqueName: \"kubernetes.io/projected/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-kube-api-access-6k78g\") pod \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\" (UID: \"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f\") "
Oct 31 03:58:42.149527 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.149237    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-envoy\") pod \"e3f1c261-eea4-45ad-ad44-b0f997723e87\" (UID: \"e3f1c261-eea4-45ad-ad44-b0f997723e87\") "
Oct 31 03:58:42.149825 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.149084    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-data" pod "e3f1c261-eea4-45ad-ad44-b0f997723e87" (UID: "e3f1c261-eea4-45ad-ad44-b0f997723e87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:42.149825 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.149204    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-workload-certs" pod "e3f1c261-eea4-45ad-ad44-b0f997723e87" (UID: "e3f1c261-eea4-45ad-ad44-b0f997723e87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:42.150025 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.149377    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istiod-ca-cert" pod "cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f" (UID: "cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:58:42.150025 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.149799    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-workload-socket" pod "cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f" (UID: "cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:42.151744 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.151719    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f1c261-eea4-45ad-ad44-b0f997723e87-istiod-ca-cert" pod "e3f1c261-eea4-45ad-ad44-b0f997723e87" (UID: "e3f1c261-eea4-45ad-ad44-b0f997723e87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:58:42.151744 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.151729    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.151939 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.151766    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.151939 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.151783    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22mr4\" (UniqueName: \"kubernetes.io/projected/27ca7445-d121-4ae9-abe7-bbf2643ab453-kube-api-access-22mr4\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.151939 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.151799    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.151939 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.151812    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.151939 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.151826    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.151939 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.151840    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.151939 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.151853    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.151939 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.151868    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.151939 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.151925    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.151939 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.151940    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.152793 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.151953    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ca7445-d121-4ae9-abe7-bbf2643ab453-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.152793 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.151967    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.152793 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.151980    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.152793 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.151993    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c96k8\" (UniqueName: \"kubernetes.io/projected/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-kube-api-access-c96k8\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.152793 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.152009    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.152793 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.152023    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.152793 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.152038    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ca7445-d121-4ae9-abe7-bbf2643ab453-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.152793 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.152052    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02c6569-f6f4-41ac-bcb1-cd42873c13b2-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.152793 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.152317    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-envoy" pod "e3f1c261-eea4-45ad-ad44-b0f997723e87" (UID: "e3f1c261-eea4-45ad-ad44-b0f997723e87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:42.152793 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.152359    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-podinfo" pod "cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f" (UID: "cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:58:42.152793 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.152483    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f1c261-eea4-45ad-ad44-b0f997723e87-kube-api-access-xsqjx" pod "e3f1c261-eea4-45ad-ad44-b0f997723e87" (UID: "e3f1c261-eea4-45ad-ad44-b0f997723e87"). InnerVolumeSpecName "kube-api-access-xsqjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:58:42.152793 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.152608    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-token" pod "e3f1c261-eea4-45ad-ad44-b0f997723e87" (UID: "e3f1c261-eea4-45ad-ad44-b0f997723e87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:58:42.153438 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.152834    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-kube-api-access-6k78g" pod "cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f" (UID: "cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f"). InnerVolumeSpecName "kube-api-access-6k78g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:58:42.153617 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.153590    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-token" pod "cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f" (UID: "cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:58:42.153994 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.153976    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-podinfo" pod "e3f1c261-eea4-45ad-ad44-b0f997723e87" (UID: "e3f1c261-eea4-45ad-ad44-b0f997723e87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:58:42.154218 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.154204    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-envoy" pod "cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f" (UID: "cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:42.253680 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.253636    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.253680 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.253668    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsqjx\" (UniqueName: \"kubernetes.io/projected/e3f1c261-eea4-45ad-ad44-b0f997723e87-kube-api-access-xsqjx\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.253680 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.253679    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.253680 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.253688    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.254364 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.253696    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.254364 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.253706    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f1c261-eea4-45ad-ad44-b0f997723e87-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.254364 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.253714    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.254364 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.253722    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k78g\" (UniqueName: \"kubernetes.io/projected/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f-kube-api-access-6k78g\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.254364 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.253729    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f1c261-eea4-45ad-ad44-b0f997723e87-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:42.295803 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.295715    4447 generic.go:356] "Generic (PLEG): container finished" podID="e3f1c261-eea4-45ad-ad44-b0f997723e87" containerID="097886db4be0667859395298df95e8ab6865a2a7aff2e8bab0da330650ae5686" exitCode=0
Oct 31 03:58:42.295803 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.295765    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"
Oct 31 03:58:42.296091 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.295810    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s" event={"ID":"e3f1c261-eea4-45ad-ad44-b0f997723e87","Type":"ContainerDied","Data":"097886db4be0667859395298df95e8ab6865a2a7aff2e8bab0da330650ae5686"}
Oct 31 03:58:42.296091 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.295875    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s" event={"ID":"e3f1c261-eea4-45ad-ad44-b0f997723e87","Type":"ContainerDied","Data":"09ac262e4f52e86585d36febb749c98d6e810765a8795965c10b988d09cc7164"}
Oct 31 03:58:42.296091 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.295929    4447 scope.go:122] "RemoveContainer" containerID="097886db4be0667859395298df95e8ab6865a2a7aff2e8bab0da330650ae5686"
Oct 31 03:58:42.297956 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.297928    4447 generic.go:356] "Generic (PLEG): container finished" podID="cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f" containerID="b250474413cfbbf3a78cd51a679c7816522ac53451fd6624685caba20cac8f6e" exitCode=0
Oct 31 03:58:42.298074 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.297987    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"
Oct 31 03:58:42.298074 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.298041    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m" event={"ID":"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f","Type":"ContainerDied","Data":"b250474413cfbbf3a78cd51a679c7816522ac53451fd6624685caba20cac8f6e"}
Oct 31 03:58:42.298234 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.298094    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m" event={"ID":"cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f","Type":"ContainerDied","Data":"72a8947be904e39f009aacc069ebb6c6554bd3f1cdbdc4eb283bf20deec49b78"}
Oct 31 03:58:42.299722 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.299696    4447 generic.go:356] "Generic (PLEG): container finished" podID="27ca7445-d121-4ae9-abe7-bbf2643ab453" containerID="9f0877ad7fe748db3532567b154c0c3b1cef30d3af87dcc3f33b034fc25175be" exitCode=0
Oct 31 03:58:42.299928 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.299720    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj" event={"ID":"27ca7445-d121-4ae9-abe7-bbf2643ab453","Type":"ContainerDied","Data":"9f0877ad7fe748db3532567b154c0c3b1cef30d3af87dcc3f33b034fc25175be"}
Oct 31 03:58:42.299928 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.299737    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"
Oct 31 03:58:42.299928 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.299762    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj" event={"ID":"27ca7445-d121-4ae9-abe7-bbf2643ab453","Type":"ContainerDied","Data":"5a1f551ba5e5ade7b156fa9871b3d47a66a82f42bada6d97520b0c52a43ebc66"}
Oct 31 03:58:42.301594 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.301571    4447 generic.go:356] "Generic (PLEG): container finished" podID="a02c6569-f6f4-41ac-bcb1-cd42873c13b2" containerID="7e483ad7884cbc2c841b46bb02a0dfbde8f6200d94708c0c38282f616d74d5b9" exitCode=0
Oct 31 03:58:42.301710 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.301616    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"
Oct 31 03:58:42.301710 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.301655    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh" event={"ID":"a02c6569-f6f4-41ac-bcb1-cd42873c13b2","Type":"ContainerDied","Data":"7e483ad7884cbc2c841b46bb02a0dfbde8f6200d94708c0c38282f616d74d5b9"}
Oct 31 03:58:42.301710 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.301703    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh" event={"ID":"a02c6569-f6f4-41ac-bcb1-cd42873c13b2","Type":"ContainerDied","Data":"0fa2a18e2d0f294c5dfb22659e1105e1cf3ff1028afc3f7d1b106c9f5e35f5dc"}
Oct 31 03:58:42.309548 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.309527    4447 scope.go:122] "RemoveContainer" containerID="097886db4be0667859395298df95e8ab6865a2a7aff2e8bab0da330650ae5686"
Oct 31 03:58:42.309874 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:58:42.309851    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"097886db4be0667859395298df95e8ab6865a2a7aff2e8bab0da330650ae5686\": container with ID starting with 097886db4be0667859395298df95e8ab6865a2a7aff2e8bab0da330650ae5686 not found: ID does not exist" containerID="097886db4be0667859395298df95e8ab6865a2a7aff2e8bab0da330650ae5686"
Oct 31 03:58:42.309976 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.309901    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097886db4be0667859395298df95e8ab6865a2a7aff2e8bab0da330650ae5686"} err="failed to get container status \"097886db4be0667859395298df95e8ab6865a2a7aff2e8bab0da330650ae5686\": rpc error: code = NotFound desc = could not find container \"097886db4be0667859395298df95e8ab6865a2a7aff2e8bab0da330650ae5686\": container with ID starting with 097886db4be0667859395298df95e8ab6865a2a7aff2e8bab0da330650ae5686 not found: ID does not exist"
Oct 31 03:58:42.309976 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.309927    4447 scope.go:122] "RemoveContainer" containerID="b250474413cfbbf3a78cd51a679c7816522ac53451fd6624685caba20cac8f6e"
Oct 31 03:58:42.320474 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.320458    4447 scope.go:122] "RemoveContainer" containerID="b250474413cfbbf3a78cd51a679c7816522ac53451fd6624685caba20cac8f6e"
Oct 31 03:58:42.320740 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:58:42.320724    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b250474413cfbbf3a78cd51a679c7816522ac53451fd6624685caba20cac8f6e\": container with ID starting with b250474413cfbbf3a78cd51a679c7816522ac53451fd6624685caba20cac8f6e not found: ID does not exist" containerID="b250474413cfbbf3a78cd51a679c7816522ac53451fd6624685caba20cac8f6e"
Oct 31 03:58:42.320783 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.320748    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b250474413cfbbf3a78cd51a679c7816522ac53451fd6624685caba20cac8f6e"} err="failed to get container status \"b250474413cfbbf3a78cd51a679c7816522ac53451fd6624685caba20cac8f6e\": rpc error: code = NotFound desc = could not find container \"b250474413cfbbf3a78cd51a679c7816522ac53451fd6624685caba20cac8f6e\": container with ID starting with b250474413cfbbf3a78cd51a679c7816522ac53451fd6624685caba20cac8f6e not found: ID does not exist"
Oct 31 03:58:42.320783 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.320765    4447 scope.go:122] "RemoveContainer" containerID="9f0877ad7fe748db3532567b154c0c3b1cef30d3af87dcc3f33b034fc25175be"
Oct 31 03:58:42.330833 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.330816    4447 scope.go:122] "RemoveContainer" containerID="9f0877ad7fe748db3532567b154c0c3b1cef30d3af87dcc3f33b034fc25175be"
Oct 31 03:58:42.331150 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:58:42.331133    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f0877ad7fe748db3532567b154c0c3b1cef30d3af87dcc3f33b034fc25175be\": container with ID starting with 9f0877ad7fe748db3532567b154c0c3b1cef30d3af87dcc3f33b034fc25175be not found: ID does not exist" containerID="9f0877ad7fe748db3532567b154c0c3b1cef30d3af87dcc3f33b034fc25175be"
Oct 31 03:58:42.331198 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.331159    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0877ad7fe748db3532567b154c0c3b1cef30d3af87dcc3f33b034fc25175be"} err="failed to get container status \"9f0877ad7fe748db3532567b154c0c3b1cef30d3af87dcc3f33b034fc25175be\": rpc error: code = NotFound desc = could not find container \"9f0877ad7fe748db3532567b154c0c3b1cef30d3af87dcc3f33b034fc25175be\": container with ID starting with 9f0877ad7fe748db3532567b154c0c3b1cef30d3af87dcc3f33b034fc25175be not found: ID does not exist"
Oct 31 03:58:42.331198 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.331178    4447 scope.go:122] "RemoveContainer" containerID="7e483ad7884cbc2c841b46bb02a0dfbde8f6200d94708c0c38282f616d74d5b9"
Oct 31 03:58:42.344410 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.344389    4447 scope.go:122] "RemoveContainer" containerID="7e483ad7884cbc2c841b46bb02a0dfbde8f6200d94708c0c38282f616d74d5b9"
Oct 31 03:58:42.344818 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:58:42.344789    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e483ad7884cbc2c841b46bb02a0dfbde8f6200d94708c0c38282f616d74d5b9\": container with ID starting with 7e483ad7884cbc2c841b46bb02a0dfbde8f6200d94708c0c38282f616d74d5b9 not found: ID does not exist" containerID="7e483ad7884cbc2c841b46bb02a0dfbde8f6200d94708c0c38282f616d74d5b9"
Oct 31 03:58:42.344989 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.344829    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e483ad7884cbc2c841b46bb02a0dfbde8f6200d94708c0c38282f616d74d5b9"} err="failed to get container status \"7e483ad7884cbc2c841b46bb02a0dfbde8f6200d94708c0c38282f616d74d5b9\": rpc error: code = NotFound desc = could not find container \"7e483ad7884cbc2c841b46bb02a0dfbde8f6200d94708c0c38282f616d74d5b9\": container with ID starting with 7e483ad7884cbc2c841b46bb02a0dfbde8f6200d94708c0c38282f616d74d5b9 not found: ID does not exist"
Oct 31 03:58:42.572785 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.572692    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"]
Oct 31 03:58:42.577440 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.577416    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-lks2s"]
Oct 31 03:58:42.610145 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.610102    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd66aaca-cbe4-414b-a056-551ed0c3f0b6" path="/var/lib/kubelet/pods/dd66aaca-cbe4-414b-a056-551ed0c3f0b6/volumes"
Oct 31 03:58:42.610747 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.610732    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3f1c261-eea4-45ad-ad44-b0f997723e87" path="/var/lib/kubelet/pods/e3f1c261-eea4-45ad-ad44-b0f997723e87/volumes"
Oct 31 03:58:42.686642 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.686610    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"]
Oct 31 03:58:42.690907 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.690865    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-q5xgh"]
Oct 31 03:58:42.801998 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.801957    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"]
Oct 31 03:58:42.809989 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.809960    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-tkh6m"]
Oct 31 03:58:42.919513 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.919482    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"]
Oct 31 03:58:42.924126 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:42.924101    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1180d818-openshift-default-7ddcf5f669-ngdfj"]
Oct 31 03:58:44.610044 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:44.610013    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ca7445-d121-4ae9-abe7-bbf2643ab453" path="/var/lib/kubelet/pods/27ca7445-d121-4ae9-abe7-bbf2643ab453/volumes"
Oct 31 03:58:44.610496 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:44.610482    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a02c6569-f6f4-41ac-bcb1-cd42873c13b2" path="/var/lib/kubelet/pods/a02c6569-f6f4-41ac-bcb1-cd42873c13b2/volumes"
Oct 31 03:58:44.610902 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:44.610873    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f" path="/var/lib/kubelet/pods/cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f/volumes"
Oct 31 03:58:48.392817 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:48.392788    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq"]
Oct 31 03:58:48.499356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:48.499324    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v"]
Oct 31 03:58:49.004031 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:58:49.003991    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:58:53.393517 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:53.393460    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq" podUID="d02acebf-6463-454b-9c79-76c9cf0709fc" containerName="payload-processing" containerID="cri-o://75b0f17d630c7ea31df0a54af6f1273ddcef4638456587653073662e47603cea" gracePeriod=25
Oct 31 03:58:53.499782 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:53.499740    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v" podUID="09f019db-a086-4f4b-b1cf-499c6aacc158" containerName="payload-pre-processing" containerID="cri-o://efb1846a9f7efd209b7615399aa53308e58ad4605d88efaa119f0533d5c10a62" gracePeriod=25
Oct 31 03:58:55.550514 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:55.550476    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv"]
Oct 31 03:58:55.653655 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:55.653625    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt"]
Oct 31 03:58:56.323608 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:56.323571    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerName="main" containerID="cri-o://7f28dc5c84cfcd5917b9ab87beb931beb5a67452082a67eca1650b8e8a028f01" gracePeriod=45
Oct 31 03:58:56.608356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:56.608333    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:58:56.645541 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:56.645510    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b8712d5-205e-4fc4-bc62-9b35dca0310a-kube-api-access-ch77c\" (UniqueName: \"kubernetes.io/projected/6b8712d5-205e-4fc4-bc62-9b35dca0310a-kube-api-access-ch77c\") pod \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\" (UID: \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\") "
Oct 31 03:58:56.645722 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:56.645636    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-model-cache\") pod \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\" (UID: \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\") "
Oct 31 03:58:56.645722 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:56.645691    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-tmp-dir\") pod \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\" (UID: \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\") "
Oct 31 03:58:56.645832 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:56.645728    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-dshm\") pod \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\" (UID: \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\") "
Oct 31 03:58:56.645832 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:56.645762    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b8712d5-205e-4fc4-bc62-9b35dca0310a-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b8712d5-205e-4fc4-bc62-9b35dca0310a-tls-certs\") pod \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\" (UID: \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\") "
Oct 31 03:58:56.645981 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:56.645926    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-model-cache" pod "6b8712d5-205e-4fc4-bc62-9b35dca0310a" (UID: "6b8712d5-205e-4fc4-bc62-9b35dca0310a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:56.646039 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:56.646010    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-home\" (UniqueName: \"kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-home\") pod \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\" (UID: \"6b8712d5-205e-4fc4-bc62-9b35dca0310a\") "
Oct 31 03:58:56.646090 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:56.646034    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-tmp-dir" pod "6b8712d5-205e-4fc4-bc62-9b35dca0310a" (UID: "6b8712d5-205e-4fc4-bc62-9b35dca0310a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:56.646268 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:56.646245    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-home" pod "6b8712d5-205e-4fc4-bc62-9b35dca0310a" (UID: "6b8712d5-205e-4fc4-bc62-9b35dca0310a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:56.646481 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:56.646465    4447 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-model-cache\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:56.646579 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:56.646489    4447 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-tmp-dir\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:56.646579 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:56.646501    4447 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-home\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:56.648040 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:56.648017    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b8712d5-205e-4fc4-bc62-9b35dca0310a-kube-api-access-ch77c" pod "6b8712d5-205e-4fc4-bc62-9b35dca0310a" (UID: "6b8712d5-205e-4fc4-bc62-9b35dca0310a"). InnerVolumeSpecName "kube-api-access-ch77c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:58:56.648148 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:56.648035    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b8712d5-205e-4fc4-bc62-9b35dca0310a-tls-certs" pod "6b8712d5-205e-4fc4-bc62-9b35dca0310a" (UID: "6b8712d5-205e-4fc4-bc62-9b35dca0310a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:58:56.648148 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:56.648096    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-dshm" pod "6b8712d5-205e-4fc4-bc62-9b35dca0310a" (UID: "6b8712d5-205e-4fc4-bc62-9b35dca0310a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:56.747673 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:56.747634    4447 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b8712d5-205e-4fc4-bc62-9b35dca0310a-dshm\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:56.747673 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:56.747670    4447 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b8712d5-205e-4fc4-bc62-9b35dca0310a-tls-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:56.747867 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:56.747685    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch77c\" (UniqueName: \"kubernetes.io/projected/6b8712d5-205e-4fc4-bc62-9b35dca0310a-kube-api-access-ch77c\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:58:57.422764 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:57.422729    4447 generic.go:356] "Generic (PLEG): container finished" podID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerID="7f28dc5c84cfcd5917b9ab87beb931beb5a67452082a67eca1650b8e8a028f01" exitCode=0
Oct 31 03:58:57.422948 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:57.422770    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"
Oct 31 03:58:57.422948 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:57.422780    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" event={"ID":"6b8712d5-205e-4fc4-bc62-9b35dca0310a","Type":"ContainerDied","Data":"7f28dc5c84cfcd5917b9ab87beb931beb5a67452082a67eca1650b8e8a028f01"}
Oct 31 03:58:57.422948 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:57.422811    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq" event={"ID":"6b8712d5-205e-4fc4-bc62-9b35dca0310a","Type":"ContainerDied","Data":"ee5999556a3007eaf78aa9614a5c13d1d567eaa94d0371bfdea464c7065cc9e8"}
Oct 31 03:58:57.422948 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:57.422829    4447 scope.go:122] "RemoveContainer" containerID="7f28dc5c84cfcd5917b9ab87beb931beb5a67452082a67eca1650b8e8a028f01"
Oct 31 03:58:57.433614 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:57.433596    4447 scope.go:122] "RemoveContainer" containerID="7f28dc5c84cfcd5917b9ab87beb931beb5a67452082a67eca1650b8e8a028f01"
Oct 31 03:58:57.433868 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:58:57.433851    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f28dc5c84cfcd5917b9ab87beb931beb5a67452082a67eca1650b8e8a028f01\": container with ID starting with 7f28dc5c84cfcd5917b9ab87beb931beb5a67452082a67eca1650b8e8a028f01 not found: ID does not exist" containerID="7f28dc5c84cfcd5917b9ab87beb931beb5a67452082a67eca1650b8e8a028f01"
Oct 31 03:58:57.433940 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:57.433875    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f28dc5c84cfcd5917b9ab87beb931beb5a67452082a67eca1650b8e8a028f01"} err="failed to get container status \"7f28dc5c84cfcd5917b9ab87beb931beb5a67452082a67eca1650b8e8a028f01\": rpc error: code = NotFound desc = could not find container \"7f28dc5c84cfcd5917b9ab87beb931beb5a67452082a67eca1650b8e8a028f01\": container with ID starting with 7f28dc5c84cfcd5917b9ab87beb931beb5a67452082a67eca1650b8e8a028f01 not found: ID does not exist"
Oct 31 03:58:57.640227 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:57.640190    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"]
Oct 31 03:58:57.690314 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:57.690239    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-b63425/distinct-2-w6-b63425-kserve-c4bd7bff8-dcmvq"]
Oct 31 03:58:58.608830 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:58:58.608795    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" path="/var/lib/kubelet/pods/6b8712d5-205e-4fc4-bc62-9b35dca0310a/volumes"
Oct 31 03:59:00.355939 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:00.353596    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq" podUID="d02acebf-6463-454b-9c79-76c9cf0709fc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:59:00.355939 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:00.353615    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v" podUID="09f019db-a086-4f4b-b1cf-499c6aacc158" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:59:00.551922 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:00.551847    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv" podUID="b07daa8e-894c-42e3-ac95-a21c71b8735c" containerName="payload-processing" containerID="cri-o://871763a11296cc4e181fa04591bc59c9c48414aae88c446ad30e0737edce9642" gracePeriod=25
Oct 31 03:59:00.654961 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:00.654817    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt" podUID="72c4ce5c-8bd0-4857-97cc-8f26437cd57c" containerName="payload-pre-processing" containerID="cri-o://1f094289fd45074247364f13a81b8d6609c8c30601e31576ddaeb5b4b5232433" gracePeriod=25
Oct 31 03:59:01.142479 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.142440    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1794214-jcvbf"]
Oct 31 03:59:01.143224 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143200    4447 cpu_manager.go:414] "containerMap: removing container" podUID="a02c6569-f6f4-41ac-bcb1-cd42873c13b2" containerName="istio-proxy"
Oct 31 03:59:01.143224 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143218    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a02c6569-f6f4-41ac-bcb1-cd42873c13b2" containerName="istio-proxy"
Oct 31 03:59:01.143392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143235    4447 cpu_manager.go:414] "containerMap: removing container" podUID="cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f" containerName="istio-proxy"
Oct 31 03:59:01.143392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143241    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f" containerName="istio-proxy"
Oct 31 03:59:01.143392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143252    4447 cpu_manager.go:414] "containerMap: removing container" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerName="main"
Oct 31 03:59:01.143392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143256    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerName="main"
Oct 31 03:59:01.143392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143270    4447 cpu_manager.go:414] "containerMap: removing container" podUID="f76a9936-58a5-4203-a1c8-9d8d51d3a190" containerName="revoke-keys"
Oct 31 03:59:01.143392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143275    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f76a9936-58a5-4203-a1c8-9d8d51d3a190" containerName="revoke-keys"
Oct 31 03:59:01.143392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143285    4447 cpu_manager.go:414] "containerMap: removing container" podUID="f3b51253-b4d7-49c9-be26-36a4e5320c82" containerName="payload-pre-processing"
Oct 31 03:59:01.143392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143289    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b51253-b4d7-49c9-be26-36a4e5320c82" containerName="payload-pre-processing"
Oct 31 03:59:01.143392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143301    4447 cpu_manager.go:414] "containerMap: removing container" podUID="27ca7445-d121-4ae9-abe7-bbf2643ab453" containerName="istio-proxy"
Oct 31 03:59:01.143392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143306    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ca7445-d121-4ae9-abe7-bbf2643ab453" containerName="istio-proxy"
Oct 31 03:59:01.143392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143313    4447 cpu_manager.go:414] "containerMap: removing container" podUID="dd66aaca-cbe4-414b-a056-551ed0c3f0b6" containerName="istio-proxy"
Oct 31 03:59:01.143392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143318    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd66aaca-cbe4-414b-a056-551ed0c3f0b6" containerName="istio-proxy"
Oct 31 03:59:01.143392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143325    4447 cpu_manager.go:414] "containerMap: removing container" podUID="f334da75-10b1-4320-9d31-4b8d428040d6" containerName="payload-pre-processing"
Oct 31 03:59:01.143392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143329    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f334da75-10b1-4320-9d31-4b8d428040d6" containerName="payload-pre-processing"
Oct 31 03:59:01.143392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143362    4447 cpu_manager.go:414] "containerMap: removing container" podUID="76378008-52e0-44da-9bf4-7c6745178755" containerName="payload-processing"
Oct 31 03:59:01.143392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143370    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76378008-52e0-44da-9bf4-7c6745178755" containerName="payload-processing"
Oct 31 03:59:01.143392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143382    4447 cpu_manager.go:414] "containerMap: removing container" podUID="24d4fdea-5469-4804-801f-3068de198f4a" containerName="istio-proxy"
Oct 31 03:59:01.143392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143387    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24d4fdea-5469-4804-801f-3068de198f4a" containerName="istio-proxy"
Oct 31 03:59:01.143392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143394    4447 cpu_manager.go:414] "containerMap: removing container" podUID="683a25bd-a64b-4a4c-8dbc-df6a1dd431aa" containerName="payload-processing"
Oct 31 03:59:01.143392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143400    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="683a25bd-a64b-4a4c-8dbc-df6a1dd431aa" containerName="payload-processing"
Oct 31 03:59:01.144114 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143407    4447 cpu_manager.go:414] "containerMap: removing container" podUID="e3f1c261-eea4-45ad-ad44-b0f997723e87" containerName="istio-proxy"
Oct 31 03:59:01.144114 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143411    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3f1c261-eea4-45ad-ad44-b0f997723e87" containerName="istio-proxy"
Oct 31 03:59:01.144114 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143507    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="a02c6569-f6f4-41ac-bcb1-cd42873c13b2" containerName="istio-proxy"
Oct 31 03:59:01.144114 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143518    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b8712d5-205e-4fc4-bc62-9b35dca0310a" containerName="main"
Oct 31 03:59:01.144114 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143524    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="cada5ab7-e6b6-45e8-85ce-53c9c4d6c10f" containerName="istio-proxy"
Oct 31 03:59:01.144114 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143532    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="f76a9936-58a5-4203-a1c8-9d8d51d3a190" containerName="revoke-keys"
Oct 31 03:59:01.144114 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143538    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b51253-b4d7-49c9-be26-36a4e5320c82" containerName="payload-pre-processing"
Oct 31 03:59:01.144114 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143543    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ca7445-d121-4ae9-abe7-bbf2643ab453" containerName="istio-proxy"
Oct 31 03:59:01.144114 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143548    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd66aaca-cbe4-414b-a056-551ed0c3f0b6" containerName="istio-proxy"
Oct 31 03:59:01.144114 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143554    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="f334da75-10b1-4320-9d31-4b8d428040d6" containerName="payload-pre-processing"
Oct 31 03:59:01.144114 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143560    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="76378008-52e0-44da-9bf4-7c6745178755" containerName="payload-processing"
Oct 31 03:59:01.144114 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143566    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="24d4fdea-5469-4804-801f-3068de198f4a" containerName="istio-proxy"
Oct 31 03:59:01.144114 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143570    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="683a25bd-a64b-4a4c-8dbc-df6a1dd431aa" containerName="payload-processing"
Oct 31 03:59:01.144114 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.143575    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3f1c261-eea4-45ad-ad44-b0f997723e87" containerName="istio-proxy"
Oct 31 03:59:01.147387 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.147367    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794214-jcvbf"
Oct 31 03:59:01.205456 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.205422    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1794214-jcvbf"]
Oct 31 03:59:01.206048 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.206019    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc56ecc-85d1-4a46-b567-8ece499dba21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1794214-jcvbf\" (UID: \"cbc56ecc-85d1-4a46-b567-8ece499dba21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794214-jcvbf"
Oct 31 03:59:01.206252 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.206232    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw2bf\" (UniqueName: \"kubernetes.io/projected/cbc56ecc-85d1-4a46-b567-8ece499dba21-kube-api-access-jw2bf\") pod \"maas-api-revoke-sub-c1794214-jcvbf\" (UID: \"cbc56ecc-85d1-4a46-b567-8ece499dba21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794214-jcvbf"
Oct 31 03:59:01.206346 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.206331    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc56ecc-85d1-4a46-b567-8ece499dba21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1794214-jcvbf\" (UID: \"cbc56ecc-85d1-4a46-b567-8ece499dba21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794214-jcvbf"
Oct 31 03:59:01.250742 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.250704    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:59:01.250923 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.250787    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 03:59:01.307967 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.307924    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc56ecc-85d1-4a46-b567-8ece499dba21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1794214-jcvbf\" (UID: \"cbc56ecc-85d1-4a46-b567-8ece499dba21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794214-jcvbf"
Oct 31 03:59:01.308166 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.308110    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc56ecc-85d1-4a46-b567-8ece499dba21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1794214-jcvbf\" (UID: \"cbc56ecc-85d1-4a46-b567-8ece499dba21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794214-jcvbf"
Oct 31 03:59:01.308329 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.308313    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw2bf\" (UniqueName: \"kubernetes.io/projected/cbc56ecc-85d1-4a46-b567-8ece499dba21-kube-api-access-jw2bf\") pod \"maas-api-revoke-sub-c1794214-jcvbf\" (UID: \"cbc56ecc-85d1-4a46-b567-8ece499dba21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794214-jcvbf"
Oct 31 03:59:01.308774 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.308749    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc56ecc-85d1-4a46-b567-8ece499dba21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1794214-jcvbf\" (UID: \"cbc56ecc-85d1-4a46-b567-8ece499dba21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794214-jcvbf"
Oct 31 03:59:01.310523 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.310500    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc56ecc-85d1-4a46-b567-8ece499dba21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1794214-jcvbf\" (UID: \"cbc56ecc-85d1-4a46-b567-8ece499dba21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794214-jcvbf"
Oct 31 03:59:01.369515 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.369482    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw2bf\" (UniqueName: \"kubernetes.io/projected/cbc56ecc-85d1-4a46-b567-8ece499dba21-kube-api-access-jw2bf\") pod \"maas-api-revoke-sub-c1794214-jcvbf\" (UID: \"cbc56ecc-85d1-4a46-b567-8ece499dba21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794214-jcvbf"
Oct 31 03:59:01.461222 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.461192    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794214-jcvbf"
Oct 31 03:59:01.853974 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.853938    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1794214-jcvbf"]
Oct 31 03:59:01.854153 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:59:01.854124    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc56ecc_85d1_4a46_b567_8ece499dba21.slice/crio-29640a1b4d9ccd1251cbeaa4d19db103752536a6c62c73773fbdd237e600b912 WatchSource:0}: Error finding container 29640a1b4d9ccd1251cbeaa4d19db103752536a6c62c73773fbdd237e600b912: Status 404 returned error can't find the container with id 29640a1b4d9ccd1251cbeaa4d19db103752536a6c62c73773fbdd237e600b912
Oct 31 03:59:01.979215 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.979176    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c"]
Oct 31 03:59:01.981982 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:01.981955    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-05a9ff-v4c5c"]
Oct 31 03:59:02.461423 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:02.461382    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794214-jcvbf" event={"ID":"cbc56ecc-85d1-4a46-b567-8ece499dba21","Type":"ContainerStarted","Data":"1539f32bbcb44b4e36c9feb563abed2c812963510f8840caaa42150a09e78fcf"}
Oct 31 03:59:02.461810 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:02.461451    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794214-jcvbf" event={"ID":"cbc56ecc-85d1-4a46-b567-8ece499dba21","Type":"ContainerStarted","Data":"29640a1b4d9ccd1251cbeaa4d19db103752536a6c62c73773fbdd237e600b912"}
Oct 31 03:59:02.579745 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:02.579688    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794214-jcvbf" podStartSLOduration=1.579674109 podStartE2EDuration="1.579674109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:59:02.576908268 +0000 UTC m=+2368.087890648" watchObservedRunningTime="2026-10-31 03:59:02.579674109 +0000 UTC m=+2368.090656485"
Oct 31 03:59:02.611795 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:02.611760    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="648d0647-a65a-4d19-b539-b52e5cfc9cf8" path="/var/lib/kubelet/pods/648d0647-a65a-4d19-b539-b52e5cfc9cf8/volumes"
Oct 31 03:59:03.470686 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:03.470648    4447 generic.go:356] "Generic (PLEG): container finished" podID="cbc56ecc-85d1-4a46-b567-8ece499dba21" containerID="1539f32bbcb44b4e36c9feb563abed2c812963510f8840caaa42150a09e78fcf" exitCode=0
Oct 31 03:59:03.471106 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:03.470736    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794214-jcvbf" event={"ID":"cbc56ecc-85d1-4a46-b567-8ece499dba21","Type":"ContainerDied","Data":"1539f32bbcb44b4e36c9feb563abed2c812963510f8840caaa42150a09e78fcf"}
Oct 31 03:59:04.640332 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:04.640305    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794214-jcvbf"
Oct 31 03:59:04.754254 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:04.754155    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbc56ecc-85d1-4a46-b567-8ece499dba21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc56ecc-85d1-4a46-b567-8ece499dba21-maas-api-openshift-ca\") pod \"cbc56ecc-85d1-4a46-b567-8ece499dba21\" (UID: \"cbc56ecc-85d1-4a46-b567-8ece499dba21\") "
Oct 31 03:59:04.754415 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:04.754274    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc56ecc-85d1-4a46-b567-8ece499dba21-kube-api-access-jw2bf\" (UniqueName: \"kubernetes.io/projected/cbc56ecc-85d1-4a46-b567-8ece499dba21-kube-api-access-jw2bf\") pod \"cbc56ecc-85d1-4a46-b567-8ece499dba21\" (UID: \"cbc56ecc-85d1-4a46-b567-8ece499dba21\") "
Oct 31 03:59:04.754415 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:04.754406    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbc56ecc-85d1-4a46-b567-8ece499dba21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc56ecc-85d1-4a46-b567-8ece499dba21-maas-api-xks-ca\") pod \"cbc56ecc-85d1-4a46-b567-8ece499dba21\" (UID: \"cbc56ecc-85d1-4a46-b567-8ece499dba21\") "
Oct 31 03:59:04.754579 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:04.754544    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc56ecc-85d1-4a46-b567-8ece499dba21-maas-api-openshift-ca" pod "cbc56ecc-85d1-4a46-b567-8ece499dba21" (UID: "cbc56ecc-85d1-4a46-b567-8ece499dba21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:04.755695 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:04.755652    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbc56ecc-85d1-4a46-b567-8ece499dba21-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:04.757279 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:04.757256    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbc56ecc-85d1-4a46-b567-8ece499dba21-maas-api-xks-ca" pod "cbc56ecc-85d1-4a46-b567-8ece499dba21" (UID: "cbc56ecc-85d1-4a46-b567-8ece499dba21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:59:04.757279 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:04.757262    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc56ecc-85d1-4a46-b567-8ece499dba21-kube-api-access-jw2bf" pod "cbc56ecc-85d1-4a46-b567-8ece499dba21" (UID: "cbc56ecc-85d1-4a46-b567-8ece499dba21"). InnerVolumeSpecName "kube-api-access-jw2bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:04.857023 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:04.856991    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbc56ecc-85d1-4a46-b567-8ece499dba21-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:04.857023 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:04.857019    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw2bf\" (UniqueName: \"kubernetes.io/projected/cbc56ecc-85d1-4a46-b567-8ece499dba21-kube-api-access-jw2bf\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:05.486710 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:05.486671    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794214-jcvbf"
Oct 31 03:59:05.486880 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:05.486703    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794214-jcvbf" event={"ID":"cbc56ecc-85d1-4a46-b567-8ece499dba21","Type":"ContainerDied","Data":"29640a1b4d9ccd1251cbeaa4d19db103752536a6c62c73773fbdd237e600b912"}
Oct 31 03:59:05.486880 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:05.486754    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29640a1b4d9ccd1251cbeaa4d19db103752536a6c62c73773fbdd237e600b912"
Oct 31 03:59:07.402570 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:07.402534    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv" podUID="b07daa8e-894c-42e3-ac95-a21c71b8735c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:59:07.403018 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:07.402534    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt" podUID="72c4ce5c-8bd0-4857-97cc-8f26437cd57c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:59:10.353740 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:10.353704    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v" podUID="09f019db-a086-4f4b-b1cf-499c6aacc158" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:59:10.354192 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:10.353702    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq" podUID="d02acebf-6463-454b-9c79-76c9cf0709fc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:59:13.590718 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:13.590680    4447 generic.go:356] "Generic (PLEG): container finished" podID="d02acebf-6463-454b-9c79-76c9cf0709fc" containerID="75b0f17d630c7ea31df0a54af6f1273ddcef4638456587653073662e47603cea" exitCode=0
Oct 31 03:59:13.591312 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:13.590920    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq" event={"ID":"d02acebf-6463-454b-9c79-76c9cf0709fc","Type":"ContainerDied","Data":"75b0f17d630c7ea31df0a54af6f1273ddcef4638456587653073662e47603cea"}
Oct 31 03:59:13.737224 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:13.736780    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq"
Oct 31 03:59:13.774310 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:13.773147    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d02acebf-6463-454b-9c79-76c9cf0709fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02acebf-6463-454b-9c79-76c9cf0709fc-plugins-config-volume\") pod \"d02acebf-6463-454b-9c79-76c9cf0709fc\" (UID: \"d02acebf-6463-454b-9c79-76c9cf0709fc\") "
Oct 31 03:59:13.774310 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:13.773273    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d02acebf-6463-454b-9c79-76c9cf0709fc-kube-api-access-7pmx2\" (UniqueName: \"kubernetes.io/projected/d02acebf-6463-454b-9c79-76c9cf0709fc-kube-api-access-7pmx2\") pod \"d02acebf-6463-454b-9c79-76c9cf0709fc\" (UID: \"d02acebf-6463-454b-9c79-76c9cf0709fc\") "
Oct 31 03:59:13.774310 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:13.774266    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d02acebf-6463-454b-9c79-76c9cf0709fc-plugins-config-volume" pod "d02acebf-6463-454b-9c79-76c9cf0709fc" (UID: "d02acebf-6463-454b-9c79-76c9cf0709fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:13.803837 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:13.803197    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02acebf-6463-454b-9c79-76c9cf0709fc-kube-api-access-7pmx2" pod "d02acebf-6463-454b-9c79-76c9cf0709fc" (UID: "d02acebf-6463-454b-9c79-76c9cf0709fc"). InnerVolumeSpecName "kube-api-access-7pmx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:13.876066 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:13.875671    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d02acebf-6463-454b-9c79-76c9cf0709fc-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:13.876066 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:13.875710    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pmx2\" (UniqueName: \"kubernetes.io/projected/d02acebf-6463-454b-9c79-76c9cf0709fc-kube-api-access-7pmx2\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:13.892909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:13.892703    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v"
Oct 31 03:59:13.977050 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:13.977013    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f019db-a086-4f4b-b1cf-499c6aacc158-kube-api-access-jh9b2\" (UniqueName: \"kubernetes.io/projected/09f019db-a086-4f4b-b1cf-499c6aacc158-kube-api-access-jh9b2\") pod \"09f019db-a086-4f4b-b1cf-499c6aacc158\" (UID: \"09f019db-a086-4f4b-b1cf-499c6aacc158\") "
Oct 31 03:59:13.977231 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:13.977203    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f019db-a086-4f4b-b1cf-499c6aacc158-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f019db-a086-4f4b-b1cf-499c6aacc158-plugins-config-volume\") pod \"09f019db-a086-4f4b-b1cf-499c6aacc158\" (UID: \"09f019db-a086-4f4b-b1cf-499c6aacc158\") "
Oct 31 03:59:13.977673 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:13.977633    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f019db-a086-4f4b-b1cf-499c6aacc158-plugins-config-volume" pod "09f019db-a086-4f4b-b1cf-499c6aacc158" (UID: "09f019db-a086-4f4b-b1cf-499c6aacc158"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:13.979437 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:13.979409    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f019db-a086-4f4b-b1cf-499c6aacc158-kube-api-access-jh9b2" pod "09f019db-a086-4f4b-b1cf-499c6aacc158" (UID: "09f019db-a086-4f4b-b1cf-499c6aacc158"). InnerVolumeSpecName "kube-api-access-jh9b2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:14.078930 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:14.078868    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f019db-a086-4f4b-b1cf-499c6aacc158-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:14.078930 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:14.078932    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh9b2\" (UniqueName: \"kubernetes.io/projected/09f019db-a086-4f4b-b1cf-499c6aacc158-kube-api-access-jh9b2\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:14.500910 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:14.498230    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"]
Oct 31 03:59:14.695970 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:14.685783    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq"
Oct 31 03:59:14.698420 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:14.698386    4447 generic.go:356] "Generic (PLEG): container finished" podID="09f019db-a086-4f4b-b1cf-499c6aacc158" containerID="efb1846a9f7efd209b7615399aa53308e58ad4605d88efaa119f0533d5c10a62" exitCode=0
Oct 31 03:59:14.700046 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:14.700026    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v"
Oct 31 03:59:14.708689 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:14.708659    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq" event={"ID":"d02acebf-6463-454b-9c79-76c9cf0709fc","Type":"ContainerDied","Data":"fd6264734c1a7197b77b5f79b736d0539951ad94a4cce8275064b29e7b85ec61"}
Oct 31 03:59:14.710099 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:14.710074    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v" event={"ID":"09f019db-a086-4f4b-b1cf-499c6aacc158","Type":"ContainerDied","Data":"efb1846a9f7efd209b7615399aa53308e58ad4605d88efaa119f0533d5c10a62"}
Oct 31 03:59:14.711000 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:14.710980    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v" event={"ID":"09f019db-a086-4f4b-b1cf-499c6aacc158","Type":"ContainerDied","Data":"765a36e820e01a9d89a780622dbf918636d9e614deab0bba6f191df915aa2b22"}
Oct 31 03:59:14.711162 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:14.710932    4447 scope.go:122] "RemoveContainer" containerID="75b0f17d630c7ea31df0a54af6f1273ddcef4638456587653073662e47603cea"
Oct 31 03:59:14.812258 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:14.805296    4447 scope.go:122] "RemoveContainer" containerID="efb1846a9f7efd209b7615399aa53308e58ad4605d88efaa119f0533d5c10a62"
Oct 31 03:59:14.866675 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:14.866637    4447 scope.go:122] "RemoveContainer" containerID="efb1846a9f7efd209b7615399aa53308e58ad4605d88efaa119f0533d5c10a62"
Oct 31 03:59:14.867187 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:59:14.867131    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb1846a9f7efd209b7615399aa53308e58ad4605d88efaa119f0533d5c10a62\": container with ID starting with efb1846a9f7efd209b7615399aa53308e58ad4605d88efaa119f0533d5c10a62 not found: ID does not exist" containerID="efb1846a9f7efd209b7615399aa53308e58ad4605d88efaa119f0533d5c10a62"
Oct 31 03:59:14.867187 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:14.867162    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb1846a9f7efd209b7615399aa53308e58ad4605d88efaa119f0533d5c10a62"} err="failed to get container status \"efb1846a9f7efd209b7615399aa53308e58ad4605d88efaa119f0533d5c10a62\": rpc error: code = NotFound desc = could not find container \"efb1846a9f7efd209b7615399aa53308e58ad4605d88efaa119f0533d5c10a62\": container with ID starting with efb1846a9f7efd209b7615399aa53308e58ad4605d88efaa119f0533d5c10a62 not found: ID does not exist"
Oct 31 03:59:14.960410 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:14.960375    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v"]
Oct 31 03:59:14.963902 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:14.963859    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-b63425-565bff8947-vmn9v"]
Oct 31 03:59:15.095920 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:15.092938    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq"]
Oct 31 03:59:15.143916 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:15.113809    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-b63425-5d6b8bb75-6qvnq"]
Oct 31 03:59:16.618771 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:16.618693    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09f019db-a086-4f4b-b1cf-499c6aacc158" path="/var/lib/kubelet/pods/09f019db-a086-4f4b-b1cf-499c6aacc158/volumes"
Oct 31 03:59:16.619159 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:16.619145    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d02acebf-6463-454b-9c79-76c9cf0709fc" path="/var/lib/kubelet/pods/d02acebf-6463-454b-9c79-76c9cf0709fc/volumes"
Oct 31 03:59:17.402548 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:17.402510    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv" podUID="b07daa8e-894c-42e3-ac95-a21c71b8735c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:59:17.402760 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:17.402519    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt" podUID="72c4ce5c-8bd0-4857-97cc-8f26437cd57c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:59:19.011101 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:59:19.011067    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:59:20.772117 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:20.772083    4447 generic.go:356] "Generic (PLEG): container finished" podID="b07daa8e-894c-42e3-ac95-a21c71b8735c" containerID="871763a11296cc4e181fa04591bc59c9c48414aae88c446ad30e0737edce9642" exitCode=0
Oct 31 03:59:20.772513 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:20.772254    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv" event={"ID":"b07daa8e-894c-42e3-ac95-a21c71b8735c","Type":"ContainerDied","Data":"871763a11296cc4e181fa04591bc59c9c48414aae88c446ad30e0737edce9642"}
Oct 31 03:59:20.826177 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:20.826111    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv"
Oct 31 03:59:20.910043 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:20.910019    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt"
Oct 31 03:59:20.934043 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:20.934014    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b07daa8e-894c-42e3-ac95-a21c71b8735c-kube-api-access-w4tvv\" (UniqueName: \"kubernetes.io/projected/b07daa8e-894c-42e3-ac95-a21c71b8735c-kube-api-access-w4tvv\") pod \"b07daa8e-894c-42e3-ac95-a21c71b8735c\" (UID: \"b07daa8e-894c-42e3-ac95-a21c71b8735c\") "
Oct 31 03:59:20.934202 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:20.934108    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b07daa8e-894c-42e3-ac95-a21c71b8735c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07daa8e-894c-42e3-ac95-a21c71b8735c-plugins-config-volume\") pod \"b07daa8e-894c-42e3-ac95-a21c71b8735c\" (UID: \"b07daa8e-894c-42e3-ac95-a21c71b8735c\") "
Oct 31 03:59:20.934453 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:20.934431    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b07daa8e-894c-42e3-ac95-a21c71b8735c-plugins-config-volume" pod "b07daa8e-894c-42e3-ac95-a21c71b8735c" (UID: "b07daa8e-894c-42e3-ac95-a21c71b8735c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:20.934569 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:20.934556    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b07daa8e-894c-42e3-ac95-a21c71b8735c-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:20.936293 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:20.936273    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b07daa8e-894c-42e3-ac95-a21c71b8735c-kube-api-access-w4tvv" pod "b07daa8e-894c-42e3-ac95-a21c71b8735c" (UID: "b07daa8e-894c-42e3-ac95-a21c71b8735c"). InnerVolumeSpecName "kube-api-access-w4tvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:21.035404 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:21.035376    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72c4ce5c-8bd0-4857-97cc-8f26437cd57c-kube-api-access-mjcc8\" (UniqueName: \"kubernetes.io/projected/72c4ce5c-8bd0-4857-97cc-8f26437cd57c-kube-api-access-mjcc8\") pod \"72c4ce5c-8bd0-4857-97cc-8f26437cd57c\" (UID: \"72c4ce5c-8bd0-4857-97cc-8f26437cd57c\") "
Oct 31 03:59:21.035570 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:21.035546    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72c4ce5c-8bd0-4857-97cc-8f26437cd57c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72c4ce5c-8bd0-4857-97cc-8f26437cd57c-plugins-config-volume\") pod \"72c4ce5c-8bd0-4857-97cc-8f26437cd57c\" (UID: \"72c4ce5c-8bd0-4857-97cc-8f26437cd57c\") "
Oct 31 03:59:21.035847 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:21.035833    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4tvv\" (UniqueName: \"kubernetes.io/projected/b07daa8e-894c-42e3-ac95-a21c71b8735c-kube-api-access-w4tvv\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:21.035991 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:21.035923    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72c4ce5c-8bd0-4857-97cc-8f26437cd57c-plugins-config-volume" pod "72c4ce5c-8bd0-4857-97cc-8f26437cd57c" (UID: "72c4ce5c-8bd0-4857-97cc-8f26437cd57c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:21.037605 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:21.037584    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c4ce5c-8bd0-4857-97cc-8f26437cd57c-kube-api-access-mjcc8" pod "72c4ce5c-8bd0-4857-97cc-8f26437cd57c" (UID: "72c4ce5c-8bd0-4857-97cc-8f26437cd57c"). InnerVolumeSpecName "kube-api-access-mjcc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:21.137473 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:21.137388    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72c4ce5c-8bd0-4857-97cc-8f26437cd57c-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:21.137473 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:21.137419    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjcc8\" (UniqueName: \"kubernetes.io/projected/72c4ce5c-8bd0-4857-97cc-8f26437cd57c-kube-api-access-mjcc8\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:21.497603 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:21.497557    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk"]
Oct 31 03:59:21.784928 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:21.784824    4447 generic.go:356] "Generic (PLEG): container finished" podID="72c4ce5c-8bd0-4857-97cc-8f26437cd57c" containerID="1f094289fd45074247364f13a81b8d6609c8c30601e31576ddaeb5b4b5232433" exitCode=0
Oct 31 03:59:21.785384 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:21.784938    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt"
Oct 31 03:59:21.785384 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:21.784930    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt" event={"ID":"72c4ce5c-8bd0-4857-97cc-8f26437cd57c","Type":"ContainerDied","Data":"1f094289fd45074247364f13a81b8d6609c8c30601e31576ddaeb5b4b5232433"}
Oct 31 03:59:21.785384 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:21.785105    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt" event={"ID":"72c4ce5c-8bd0-4857-97cc-8f26437cd57c","Type":"ContainerDied","Data":"5e9e58a3ea630604bca7c2e7c2f13a3047f3618197ef0575589f471adb5ab669"}
Oct 31 03:59:21.785384 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:21.785152    4447 scope.go:122] "RemoveContainer" containerID="1f094289fd45074247364f13a81b8d6609c8c30601e31576ddaeb5b4b5232433"
Oct 31 03:59:21.787246 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:21.787227    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv"
Oct 31 03:59:21.787425 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:21.787397    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv" event={"ID":"b07daa8e-894c-42e3-ac95-a21c71b8735c","Type":"ContainerDied","Data":"6fbc2b4e79c5f45734fc9280f1002fbadb9fb086e325c76666ef1bbf8fb5012c"}
Oct 31 03:59:21.799507 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:21.799483    4447 scope.go:122] "RemoveContainer" containerID="1f094289fd45074247364f13a81b8d6609c8c30601e31576ddaeb5b4b5232433"
Oct 31 03:59:21.799860 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:59:21.799838    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f094289fd45074247364f13a81b8d6609c8c30601e31576ddaeb5b4b5232433\": container with ID starting with 1f094289fd45074247364f13a81b8d6609c8c30601e31576ddaeb5b4b5232433 not found: ID does not exist" containerID="1f094289fd45074247364f13a81b8d6609c8c30601e31576ddaeb5b4b5232433"
Oct 31 03:59:21.800027 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:21.799873    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f094289fd45074247364f13a81b8d6609c8c30601e31576ddaeb5b4b5232433"} err="failed to get container status \"1f094289fd45074247364f13a81b8d6609c8c30601e31576ddaeb5b4b5232433\": rpc error: code = NotFound desc = could not find container \"1f094289fd45074247364f13a81b8d6609c8c30601e31576ddaeb5b4b5232433\": container with ID starting with 1f094289fd45074247364f13a81b8d6609c8c30601e31576ddaeb5b4b5232433 not found: ID does not exist"
Oct 31 03:59:21.800027 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:21.799950    4447 scope.go:122] "RemoveContainer" containerID="871763a11296cc4e181fa04591bc59c9c48414aae88c446ad30e0737edce9642"
Oct 31 03:59:21.954954 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:21.954925    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv"]
Oct 31 03:59:21.960782 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:21.960754    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-b98893-fdd5cfbf9-8sxkv"]
Oct 31 03:59:22.074183 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:22.074106    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt"]
Oct 31 03:59:22.080336 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:22.080302    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b98893-6dfc4d9fcf-ccgjt"]
Oct 31 03:59:22.611110 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:22.611073    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72c4ce5c-8bd0-4857-97cc-8f26437cd57c" path="/var/lib/kubelet/pods/72c4ce5c-8bd0-4857-97cc-8f26437cd57c/volumes"
Oct 31 03:59:22.611619 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:22.611604    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b07daa8e-894c-42e3-ac95-a21c71b8735c" path="/var/lib/kubelet/pods/b07daa8e-894c-42e3-ac95-a21c71b8735c/volumes"
Oct 31 03:59:26.498249 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:26.498211    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk" podUID="de5b552a-3495-40e0-8f75-8fd08752474e" containerName="payload-processing" containerID="cri-o://dbfec1b33592296f28be1ee9efa702406eeea9560bbd2ddd44930425ebf498bc" gracePeriod=25
Oct 31 03:59:27.553228 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:27.553189    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk" podUID="de5b552a-3495-40e0-8f75-8fd08752474e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:59:29.669380 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:29.669339    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main" containerID="cri-o://c1b9ff58d3120c3442e2c679dabdab4d79993a84610858c8be1287bd4103a006" gracePeriod=45
Oct 31 03:59:29.863350 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:29.863061    4447 generic.go:356] "Generic (PLEG): container finished" podID="00e44046-fb2c-481f-8221-190358d77d1a" containerID="c1b9ff58d3120c3442e2c679dabdab4d79993a84610858c8be1287bd4103a006" exitCode=0
Oct 31 03:59:29.863350 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:29.863291    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" event={"ID":"00e44046-fb2c-481f-8221-190358d77d1a","Type":"ContainerDied","Data":"c1b9ff58d3120c3442e2c679dabdab4d79993a84610858c8be1287bd4103a006"}
Oct 31 03:59:29.930982 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:29.930921    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:59:30.050267 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:30.050235    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5190882d-fhcqv"]
Oct 31 03:59:30.061945 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:30.061906    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-tmp-dir\") pod \"00e44046-fb2c-481f-8221-190358d77d1a\" (UID: \"00e44046-fb2c-481f-8221-190358d77d1a\") "
Oct 31 03:59:30.062143 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:30.061996    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-model-cache\") pod \"00e44046-fb2c-481f-8221-190358d77d1a\" (UID: \"00e44046-fb2c-481f-8221-190358d77d1a\") "
Oct 31 03:59:30.062143 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:30.062136    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-home\" (UniqueName: \"kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-home\") pod \"00e44046-fb2c-481f-8221-190358d77d1a\" (UID: \"00e44046-fb2c-481f-8221-190358d77d1a\") "
Oct 31 03:59:30.062269 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:30.062203    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-tmp-dir" pod "00e44046-fb2c-481f-8221-190358d77d1a" (UID: "00e44046-fb2c-481f-8221-190358d77d1a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:30.062330 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:30.062273    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-model-cache" pod "00e44046-fb2c-481f-8221-190358d77d1a" (UID: "00e44046-fb2c-481f-8221-190358d77d1a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:30.062405 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:30.062384    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-home" pod "00e44046-fb2c-481f-8221-190358d77d1a" (UID: "00e44046-fb2c-481f-8221-190358d77d1a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:30.062455 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:30.062439    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00e44046-fb2c-481f-8221-190358d77d1a-kube-api-access-vfppc\" (UniqueName: \"kubernetes.io/projected/00e44046-fb2c-481f-8221-190358d77d1a-kube-api-access-vfppc\") pod \"00e44046-fb2c-481f-8221-190358d77d1a\" (UID: \"00e44046-fb2c-481f-8221-190358d77d1a\") "
Oct 31 03:59:30.062524 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:30.062509    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00e44046-fb2c-481f-8221-190358d77d1a-tls-certs\" (UniqueName: \"kubernetes.io/secret/00e44046-fb2c-481f-8221-190358d77d1a-tls-certs\") pod \"00e44046-fb2c-481f-8221-190358d77d1a\" (UID: \"00e44046-fb2c-481f-8221-190358d77d1a\") "
Oct 31 03:59:30.062615 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:30.062584    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-dshm\") pod \"00e44046-fb2c-481f-8221-190358d77d1a\" (UID: \"00e44046-fb2c-481f-8221-190358d77d1a\") "
Oct 31 03:59:30.068590 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:30.065765    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e44046-fb2c-481f-8221-190358d77d1a-kube-api-access-vfppc" pod "00e44046-fb2c-481f-8221-190358d77d1a" (UID: "00e44046-fb2c-481f-8221-190358d77d1a"). InnerVolumeSpecName "kube-api-access-vfppc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:30.068590 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:30.068406    4447 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-home\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:30.068590 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:30.068432    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfppc\" (UniqueName: \"kubernetes.io/projected/00e44046-fb2c-481f-8221-190358d77d1a-kube-api-access-vfppc\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:30.068590 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:30.068446    4447 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-tmp-dir\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:30.068590 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:30.068510    4447 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-model-cache\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:30.069392 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:30.069358    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00e44046-fb2c-481f-8221-190358d77d1a-tls-certs" pod "00e44046-fb2c-481f-8221-190358d77d1a" (UID: "00e44046-fb2c-481f-8221-190358d77d1a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:59:30.069864 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:30.069832    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-dshm" pod "00e44046-fb2c-481f-8221-190358d77d1a" (UID: "00e44046-fb2c-481f-8221-190358d77d1a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:30.104351 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:30.104322    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5190882d-fhcqv"]
Oct 31 03:59:30.169536 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:30.169497    4447 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e44046-fb2c-481f-8221-190358d77d1a-dshm\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:30.169536 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:30.169527    4447 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e44046-fb2c-481f-8221-190358d77d1a-tls-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:30.610955 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:30.610922    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d763a0-757d-4e92-afbb-25c2a839ef0b" path="/var/lib/kubelet/pods/f1d763a0-757d-4e92-afbb-25c2a839ef0b/volumes"
Oct 31 03:59:30.871757 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:30.871670    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"
Oct 31 03:59:30.871757 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:30.871706    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48" event={"ID":"00e44046-fb2c-481f-8221-190358d77d1a","Type":"ContainerDied","Data":"978df90d85f49d0a65ca74dab3da2f2e4c73620704c7f3027dafa70e9a4d5575"}
Oct 31 03:59:30.871757 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:30.871755    4447 scope.go:122] "RemoveContainer" containerID="c1b9ff58d3120c3442e2c679dabdab4d79993a84610858c8be1287bd4103a006"
Oct 31 03:59:31.038427 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:31.038392    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"]
Oct 31 03:59:31.087979 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:31.087948    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-909365/distinct-w0-909365-kserve-85796b5977-68h48"]
Oct 31 03:59:32.608782 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:32.608752    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00e44046-fb2c-481f-8221-190358d77d1a" path="/var/lib/kubelet/pods/00e44046-fb2c-481f-8221-190358d77d1a/volumes"
Oct 31 03:59:33.832474 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:33.832438    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"]
Oct 31 03:59:33.832949 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:33.832621    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2" podUID="da528f3a-515b-4c82-b113-aed67ef952db" containerName="maas-api" containerID="cri-o://5a73cb7d4ccaf9c91670a59969cffec42fc99d83d0a6205e7122e64e03c53844" gracePeriod=30
Oct 31 03:59:34.120483 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:34.120459    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"
Oct 31 03:59:34.231532 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:34.231496    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da528f3a-515b-4c82-b113-aed67ef952db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da528f3a-515b-4c82-b113-aed67ef952db-openshift-service-ca\") pod \"da528f3a-515b-4c82-b113-aed67ef952db\" (UID: \"da528f3a-515b-4c82-b113-aed67ef952db\") "
Oct 31 03:59:34.231739 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:34.231575    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da528f3a-515b-4c82-b113-aed67ef952db-kube-api-access-4669l\" (UniqueName: \"kubernetes.io/projected/da528f3a-515b-4c82-b113-aed67ef952db-kube-api-access-4669l\") pod \"da528f3a-515b-4c82-b113-aed67ef952db\" (UID: \"da528f3a-515b-4c82-b113-aed67ef952db\") "
Oct 31 03:59:34.231739 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:34.231606    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da528f3a-515b-4c82-b113-aed67ef952db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da528f3a-515b-4c82-b113-aed67ef952db-maas-api-tls\") pod \"da528f3a-515b-4c82-b113-aed67ef952db\" (UID: \"da528f3a-515b-4c82-b113-aed67ef952db\") "
Oct 31 03:59:34.231739 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:34.231711    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da528f3a-515b-4c82-b113-aed67ef952db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da528f3a-515b-4c82-b113-aed67ef952db-metrics-certs\") pod \"da528f3a-515b-4c82-b113-aed67ef952db\" (UID: \"da528f3a-515b-4c82-b113-aed67ef952db\") "
Oct 31 03:59:34.232053 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:34.232018    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da528f3a-515b-4c82-b113-aed67ef952db-openshift-service-ca" pod "da528f3a-515b-4c82-b113-aed67ef952db" (UID: "da528f3a-515b-4c82-b113-aed67ef952db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:34.232343 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:34.232317    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da528f3a-515b-4c82-b113-aed67ef952db-openshift-service-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:34.233809 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:34.233788    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da528f3a-515b-4c82-b113-aed67ef952db-maas-api-tls" pod "da528f3a-515b-4c82-b113-aed67ef952db" (UID: "da528f3a-515b-4c82-b113-aed67ef952db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:59:34.233945 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:34.233925    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da528f3a-515b-4c82-b113-aed67ef952db-metrics-certs" pod "da528f3a-515b-4c82-b113-aed67ef952db" (UID: "da528f3a-515b-4c82-b113-aed67ef952db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:59:34.234009 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:34.233971    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da528f3a-515b-4c82-b113-aed67ef952db-kube-api-access-4669l" pod "da528f3a-515b-4c82-b113-aed67ef952db" (UID: "da528f3a-515b-4c82-b113-aed67ef952db"). InnerVolumeSpecName "kube-api-access-4669l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:34.333414 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:34.333376    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4669l\" (UniqueName: \"kubernetes.io/projected/da528f3a-515b-4c82-b113-aed67ef952db-kube-api-access-4669l\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:34.333414 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:34.333414    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da528f3a-515b-4c82-b113-aed67ef952db-maas-api-tls\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:34.333614 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:34.333431    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da528f3a-515b-4c82-b113-aed67ef952db-metrics-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:34.905641 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:34.905556    4447 generic.go:356] "Generic (PLEG): container finished" podID="da528f3a-515b-4c82-b113-aed67ef952db" containerID="5a73cb7d4ccaf9c91670a59969cffec42fc99d83d0a6205e7122e64e03c53844" exitCode=0
Oct 31 03:59:34.905641 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:34.905618    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"
Oct 31 03:59:34.905641 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:34.905633    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2" event={"ID":"da528f3a-515b-4c82-b113-aed67ef952db","Type":"ContainerDied","Data":"5a73cb7d4ccaf9c91670a59969cffec42fc99d83d0a6205e7122e64e03c53844"}
Oct 31 03:59:34.906246 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:34.905685    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2" event={"ID":"da528f3a-515b-4c82-b113-aed67ef952db","Type":"ContainerDied","Data":"5d5d8f7802147fb61c4e363fb67ff0d17f9b80580b4f42766eb7f4270c57ad89"}
Oct 31 03:59:34.906246 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:34.905707    4447 scope.go:122] "RemoveContainer" containerID="5a73cb7d4ccaf9c91670a59969cffec42fc99d83d0a6205e7122e64e03c53844"
Oct 31 03:59:34.915845 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:34.915826    4447 scope.go:122] "RemoveContainer" containerID="5a73cb7d4ccaf9c91670a59969cffec42fc99d83d0a6205e7122e64e03c53844"
Oct 31 03:59:34.916192 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:59:34.916174    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a73cb7d4ccaf9c91670a59969cffec42fc99d83d0a6205e7122e64e03c53844\": container with ID starting with 5a73cb7d4ccaf9c91670a59969cffec42fc99d83d0a6205e7122e64e03c53844 not found: ID does not exist" containerID="5a73cb7d4ccaf9c91670a59969cffec42fc99d83d0a6205e7122e64e03c53844"
Oct 31 03:59:34.916249 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:34.916202    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a73cb7d4ccaf9c91670a59969cffec42fc99d83d0a6205e7122e64e03c53844"} err="failed to get container status \"5a73cb7d4ccaf9c91670a59969cffec42fc99d83d0a6205e7122e64e03c53844\": rpc error: code = NotFound desc = could not find container \"5a73cb7d4ccaf9c91670a59969cffec42fc99d83d0a6205e7122e64e03c53844\": container with ID starting with 5a73cb7d4ccaf9c91670a59969cffec42fc99d83d0a6205e7122e64e03c53844 not found: ID does not exist"
Oct 31 03:59:35.016849 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:35.016823    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 03:59:35.037089 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:35.037063    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 03:59:35.071513 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:35.071475    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"]
Oct 31 03:59:35.121390 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:35.121354    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-b63425-55cff9fc45-m2kh2"]
Oct 31 03:59:36.608627 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:36.608592    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da528f3a-515b-4c82-b113-aed67ef952db" path="/var/lib/kubelet/pods/da528f3a-515b-4c82-b113-aed67ef952db/volumes"
Oct 31 03:59:36.848403 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:36.848372    4447 scope.go:122] "RemoveContainer" containerID="d03beac4c3f556b9e90b4535dcd1354442b991728fa67184389b5db869501e6e"
Oct 31 03:59:36.859178 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:36.859157    4447 scope.go:122] "RemoveContainer" containerID="1c441a63dacf0748f040570cf32a76fa8109693ce7430e475e695978551aca87"
Oct 31 03:59:36.869754 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:36.869728    4447 scope.go:122] "RemoveContainer" containerID="ed78b34a0860a7f5b0e1019015bfe3c75157ae062976a9fa45d6a8d34d5180e5"
Oct 31 03:59:36.879894 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:36.879855    4447 scope.go:122] "RemoveContainer" containerID="460dda9937a8e041e6bbefeae2c8bbda525e0d4635938555bf3ac5486dad9cc4"
Oct 31 03:59:37.553569 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:37.553528    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk" podUID="de5b552a-3495-40e0-8f75-8fd08752474e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:59:43.064879 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:43.064846    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"]
Oct 31 03:59:43.065303 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:43.065069    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg" podUID="c29a47d5-b793-43c1-befc-b32b16e31114" containerName="istio-proxy" containerID="cri-o://903b2472e6a9e553e5dd85d59116f24c8e730caed53d6b112868f64b953f71c9" gracePeriod=30
Oct 31 03:59:43.749735 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:43.749702    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp"]
Oct 31 03:59:43.859371 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:43.859339    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt"]
Oct 31 03:59:44.725223 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:44.725185    4447 patch_prober.go:29] interesting pod/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:59:44.725757 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:44.725256    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg" podUID="c29a47d5-b793-43c1-befc-b32b16e31114" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:59:46.763281 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:46.763216    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk"
Oct 31 03:59:46.895764 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:46.895734    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de5b552a-3495-40e0-8f75-8fd08752474e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de5b552a-3495-40e0-8f75-8fd08752474e-plugins-config-volume\") pod \"de5b552a-3495-40e0-8f75-8fd08752474e\" (UID: \"de5b552a-3495-40e0-8f75-8fd08752474e\") "
Oct 31 03:59:46.895962 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:46.895814    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de5b552a-3495-40e0-8f75-8fd08752474e-kube-api-access-l95fq\" (UniqueName: \"kubernetes.io/projected/de5b552a-3495-40e0-8f75-8fd08752474e-kube-api-access-l95fq\") pod \"de5b552a-3495-40e0-8f75-8fd08752474e\" (UID: \"de5b552a-3495-40e0-8f75-8fd08752474e\") "
Oct 31 03:59:46.896176 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:46.896150    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de5b552a-3495-40e0-8f75-8fd08752474e-plugins-config-volume" pod "de5b552a-3495-40e0-8f75-8fd08752474e" (UID: "de5b552a-3495-40e0-8f75-8fd08752474e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:46.896430 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:46.896413    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de5b552a-3495-40e0-8f75-8fd08752474e-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:46.898094 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:46.898065    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5b552a-3495-40e0-8f75-8fd08752474e-kube-api-access-l95fq" pod "de5b552a-3495-40e0-8f75-8fd08752474e" (UID: "de5b552a-3495-40e0-8f75-8fd08752474e"). InnerVolumeSpecName "kube-api-access-l95fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:47.005233 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:47.005195    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l95fq\" (UniqueName: \"kubernetes.io/projected/de5b552a-3495-40e0-8f75-8fd08752474e-kube-api-access-l95fq\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:47.007625 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:47.007594    4447 generic.go:356] "Generic (PLEG): container finished" podID="de5b552a-3495-40e0-8f75-8fd08752474e" containerID="dbfec1b33592296f28be1ee9efa702406eeea9560bbd2ddd44930425ebf498bc" exitCode=0
Oct 31 03:59:47.007782 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:47.007668    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk"
Oct 31 03:59:47.007782 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:47.007682    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk" event={"ID":"de5b552a-3495-40e0-8f75-8fd08752474e","Type":"ContainerDied","Data":"dbfec1b33592296f28be1ee9efa702406eeea9560bbd2ddd44930425ebf498bc"}
Oct 31 03:59:47.007782 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:47.007765    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk" event={"ID":"de5b552a-3495-40e0-8f75-8fd08752474e","Type":"ContainerDied","Data":"02a69a8145d10f9777f88018b8a0cdf9e4374fc53317214d96e7bdf9f07098bf"}
Oct 31 03:59:47.007944 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:47.007791    4447 scope.go:122] "RemoveContainer" containerID="dbfec1b33592296f28be1ee9efa702406eeea9560bbd2ddd44930425ebf498bc"
Oct 31 03:59:47.018528 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:47.018307    4447 scope.go:122] "RemoveContainer" containerID="dbfec1b33592296f28be1ee9efa702406eeea9560bbd2ddd44930425ebf498bc"
Oct 31 03:59:47.018995 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:59:47.018965    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbfec1b33592296f28be1ee9efa702406eeea9560bbd2ddd44930425ebf498bc\": container with ID starting with dbfec1b33592296f28be1ee9efa702406eeea9560bbd2ddd44930425ebf498bc not found: ID does not exist" containerID="dbfec1b33592296f28be1ee9efa702406eeea9560bbd2ddd44930425ebf498bc"
Oct 31 03:59:47.019103 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:47.019007    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbfec1b33592296f28be1ee9efa702406eeea9560bbd2ddd44930425ebf498bc"} err="failed to get container status \"dbfec1b33592296f28be1ee9efa702406eeea9560bbd2ddd44930425ebf498bc\": rpc error: code = NotFound desc = could not find container \"dbfec1b33592296f28be1ee9efa702406eeea9560bbd2ddd44930425ebf498bc\": container with ID starting with dbfec1b33592296f28be1ee9efa702406eeea9560bbd2ddd44930425ebf498bc not found: ID does not exist"
Oct 31 03:59:47.173814 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:47.173781    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk"]
Oct 31 03:59:47.180490 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:47.180462    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-909365-5f867796ff-jb9pk"]
Oct 31 03:59:48.327090 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.327059    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:59:48.420113 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.420021    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c29a47d5-b793-43c1-befc-b32b16e31114-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29a47d5-b793-43c1-befc-b32b16e31114-istio-podinfo\") pod \"c29a47d5-b793-43c1-befc-b32b16e31114\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") "
Oct 31 03:59:48.420113 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.420074    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c29a47d5-b793-43c1-befc-b32b16e31114-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29a47d5-b793-43c1-befc-b32b16e31114-istiod-ca-cert\") pod \"c29a47d5-b793-43c1-befc-b32b16e31114\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") "
Oct 31 03:59:48.420388 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.420160    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c29a47d5-b793-43c1-befc-b32b16e31114-istio-token\" (UniqueName: \"kubernetes.io/projected/c29a47d5-b793-43c1-befc-b32b16e31114-istio-token\") pod \"c29a47d5-b793-43c1-befc-b32b16e31114\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") "
Oct 31 03:59:48.420388 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.420231    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c29a47d5-b793-43c1-befc-b32b16e31114-kube-api-access-wwgts\" (UniqueName: \"kubernetes.io/projected/c29a47d5-b793-43c1-befc-b32b16e31114-kube-api-access-wwgts\") pod \"c29a47d5-b793-43c1-befc-b32b16e31114\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") "
Oct 31 03:59:48.420388 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.420310    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-workload-certs\") pod \"c29a47d5-b793-43c1-befc-b32b16e31114\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") "
Oct 31 03:59:48.420388 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.420347    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-istio-data\") pod \"c29a47d5-b793-43c1-befc-b32b16e31114\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") "
Oct 31 03:59:48.420595 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.420426    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-istio-envoy\") pod \"c29a47d5-b793-43c1-befc-b32b16e31114\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") "
Oct 31 03:59:48.420595 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.420475    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-workload-socket\") pod \"c29a47d5-b793-43c1-befc-b32b16e31114\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") "
Oct 31 03:59:48.420595 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.420518    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-credential-socket\") pod \"c29a47d5-b793-43c1-befc-b32b16e31114\" (UID: \"c29a47d5-b793-43c1-befc-b32b16e31114\") "
Oct 31 03:59:48.420755 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.420616    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-istio-data" pod "c29a47d5-b793-43c1-befc-b32b16e31114" (UID: "c29a47d5-b793-43c1-befc-b32b16e31114"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:48.420755 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.420656    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c29a47d5-b793-43c1-befc-b32b16e31114-istiod-ca-cert" pod "c29a47d5-b793-43c1-befc-b32b16e31114" (UID: "c29a47d5-b793-43c1-befc-b32b16e31114"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:48.420872 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.420852    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-workload-socket" pod "c29a47d5-b793-43c1-befc-b32b16e31114" (UID: "c29a47d5-b793-43c1-befc-b32b16e31114"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:48.420959 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.420878    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-workload-certs" pod "c29a47d5-b793-43c1-befc-b32b16e31114" (UID: "c29a47d5-b793-43c1-befc-b32b16e31114"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:48.421321 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.421221    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-credential-socket" pod "c29a47d5-b793-43c1-befc-b32b16e31114" (UID: "c29a47d5-b793-43c1-befc-b32b16e31114"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:48.421321 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.421252    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:48.421321 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.421274    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:48.421321 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.421289    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:48.421321 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.421303    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29a47d5-b793-43c1-befc-b32b16e31114-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:48.422654 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.422624    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29a47d5-b793-43c1-befc-b32b16e31114-istio-token" pod "c29a47d5-b793-43c1-befc-b32b16e31114" (UID: "c29a47d5-b793-43c1-befc-b32b16e31114"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:48.422751 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.422729    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c29a47d5-b793-43c1-befc-b32b16e31114-istio-podinfo" pod "c29a47d5-b793-43c1-befc-b32b16e31114" (UID: "c29a47d5-b793-43c1-befc-b32b16e31114"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:59:48.422797 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.422743    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29a47d5-b793-43c1-befc-b32b16e31114-kube-api-access-wwgts" pod "c29a47d5-b793-43c1-befc-b32b16e31114" (UID: "c29a47d5-b793-43c1-befc-b32b16e31114"). InnerVolumeSpecName "kube-api-access-wwgts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:48.422958 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.422933    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-istio-envoy" pod "c29a47d5-b793-43c1-befc-b32b16e31114" (UID: "c29a47d5-b793-43c1-befc-b32b16e31114"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:48.522718 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.522686    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29a47d5-b793-43c1-befc-b32b16e31114-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:48.522718 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.522716    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29a47d5-b793-43c1-befc-b32b16e31114-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:48.522966 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.522725    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwgts\" (UniqueName: \"kubernetes.io/projected/c29a47d5-b793-43c1-befc-b32b16e31114-kube-api-access-wwgts\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:48.522966 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.522736    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:48.522966 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.522748    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29a47d5-b793-43c1-befc-b32b16e31114-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:48.609546 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.609510    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de5b552a-3495-40e0-8f75-8fd08752474e" path="/var/lib/kubelet/pods/de5b552a-3495-40e0-8f75-8fd08752474e/volumes"
Oct 31 03:59:48.750998 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.750948    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp" podUID="b6d42eac-1761-4ff4-a92c-9de57c7fb84a" containerName="payload-processing" containerID="cri-o://113abeae90c9a322ae0507db8d9255c748f61c3a61a0f66ae09511f641eb27f1" gracePeriod=25
Oct 31 03:59:48.859978 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:48.859900    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt" podUID="22c3bb42-e607-495f-942f-15072c8a0cf6" containerName="payload-pre-processing" containerID="cri-o://072c8bc8d8671e6930444bb66cb620e0ca4d3c38b8b9be366fc6962df2268eed" gracePeriod=25
Oct 31 03:59:49.002628 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:59:49.002547    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 03:59:49.023009 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:49.022979    4447 generic.go:356] "Generic (PLEG): container finished" podID="c29a47d5-b793-43c1-befc-b32b16e31114" containerID="903b2472e6a9e553e5dd85d59116f24c8e730caed53d6b112868f64b953f71c9" exitCode=0
Oct 31 03:59:49.023183 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:49.023026    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"
Oct 31 03:59:49.023183 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:49.023051    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg" event={"ID":"c29a47d5-b793-43c1-befc-b32b16e31114","Type":"ContainerDied","Data":"903b2472e6a9e553e5dd85d59116f24c8e730caed53d6b112868f64b953f71c9"}
Oct 31 03:59:49.023183 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:49.023104    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg" event={"ID":"c29a47d5-b793-43c1-befc-b32b16e31114","Type":"ContainerDied","Data":"7cd930df61870eeb80720e2bedccf97598d828b8706f21e013c501bf7281dfeb"}
Oct 31 03:59:49.023183 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:49.023127    4447 scope.go:122] "RemoveContainer" containerID="903b2472e6a9e553e5dd85d59116f24c8e730caed53d6b112868f64b953f71c9"
Oct 31 03:59:49.033148 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:49.033122    4447 scope.go:122] "RemoveContainer" containerID="903b2472e6a9e553e5dd85d59116f24c8e730caed53d6b112868f64b953f71c9"
Oct 31 03:59:49.033437 ip-10-0-134-255 kubenswrapper[4447]: E1031 03:59:49.033419    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"903b2472e6a9e553e5dd85d59116f24c8e730caed53d6b112868f64b953f71c9\": container with ID starting with 903b2472e6a9e553e5dd85d59116f24c8e730caed53d6b112868f64b953f71c9 not found: ID does not exist" containerID="903b2472e6a9e553e5dd85d59116f24c8e730caed53d6b112868f64b953f71c9"
Oct 31 03:59:49.033488 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:49.033449    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"903b2472e6a9e553e5dd85d59116f24c8e730caed53d6b112868f64b953f71c9"} err="failed to get container status \"903b2472e6a9e553e5dd85d59116f24c8e730caed53d6b112868f64b953f71c9\": rpc error: code = NotFound desc = could not find container \"903b2472e6a9e553e5dd85d59116f24c8e730caed53d6b112868f64b953f71c9\": container with ID starting with 903b2472e6a9e553e5dd85d59116f24c8e730caed53d6b112868f64b953f71c9 not found: ID does not exist"
Oct 31 03:59:49.190454 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:49.190416    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"]
Oct 31 03:59:49.194198 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:49.194174    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-b63425-openshift-default-7df6f8bff7-7bbxg"]
Oct 31 03:59:50.609938 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.609781    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c29a47d5-b793-43c1-befc-b32b16e31114" path="/var/lib/kubelet/pods/c29a47d5-b793-43c1-befc-b32b16e31114/volumes"
Oct 31 03:59:50.851436 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.851397    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4418c521-p8kzs"]
Oct 31 03:59:50.852175 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852154    4447 cpu_manager.go:414] "containerMap: removing container" podUID="da528f3a-515b-4c82-b113-aed67ef952db" containerName="maas-api"
Oct 31 03:59:50.852175 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852175    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da528f3a-515b-4c82-b113-aed67ef952db" containerName="maas-api"
Oct 31 03:59:50.852356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852194    4447 cpu_manager.go:414] "containerMap: removing container" podUID="de5b552a-3495-40e0-8f75-8fd08752474e" containerName="payload-processing"
Oct 31 03:59:50.852356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852200    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5b552a-3495-40e0-8f75-8fd08752474e" containerName="payload-processing"
Oct 31 03:59:50.852356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852215    4447 cpu_manager.go:414] "containerMap: removing container" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main"
Oct 31 03:59:50.852356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852221    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main"
Oct 31 03:59:50.852356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852230    4447 cpu_manager.go:414] "containerMap: removing container" podUID="c29a47d5-b793-43c1-befc-b32b16e31114" containerName="istio-proxy"
Oct 31 03:59:50.852356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852236    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c29a47d5-b793-43c1-befc-b32b16e31114" containerName="istio-proxy"
Oct 31 03:59:50.852356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852246    4447 cpu_manager.go:414] "containerMap: removing container" podUID="09f019db-a086-4f4b-b1cf-499c6aacc158" containerName="payload-pre-processing"
Oct 31 03:59:50.852356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852252    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f019db-a086-4f4b-b1cf-499c6aacc158" containerName="payload-pre-processing"
Oct 31 03:59:50.852356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852263    4447 cpu_manager.go:414] "containerMap: removing container" podUID="b07daa8e-894c-42e3-ac95-a21c71b8735c" containerName="payload-processing"
Oct 31 03:59:50.852356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852271    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b07daa8e-894c-42e3-ac95-a21c71b8735c" containerName="payload-processing"
Oct 31 03:59:50.852356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852282    4447 cpu_manager.go:414] "containerMap: removing container" podUID="cbc56ecc-85d1-4a46-b567-8ece499dba21" containerName="revoke-keys"
Oct 31 03:59:50.852356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852286    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc56ecc-85d1-4a46-b567-8ece499dba21" containerName="revoke-keys"
Oct 31 03:59:50.852356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852309    4447 cpu_manager.go:414] "containerMap: removing container" podUID="72c4ce5c-8bd0-4857-97cc-8f26437cd57c" containerName="payload-pre-processing"
Oct 31 03:59:50.852356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852313    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72c4ce5c-8bd0-4857-97cc-8f26437cd57c" containerName="payload-pre-processing"
Oct 31 03:59:50.852356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852325    4447 cpu_manager.go:414] "containerMap: removing container" podUID="d02acebf-6463-454b-9c79-76c9cf0709fc" containerName="payload-processing"
Oct 31 03:59:50.852356 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852329    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d02acebf-6463-454b-9c79-76c9cf0709fc" containerName="payload-processing"
Oct 31 03:59:50.852962 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852413    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="72c4ce5c-8bd0-4857-97cc-8f26437cd57c" containerName="payload-pre-processing"
Oct 31 03:59:50.852962 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852424    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="d02acebf-6463-454b-9c79-76c9cf0709fc" containerName="payload-processing"
Oct 31 03:59:50.852962 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852436    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="da528f3a-515b-4c82-b113-aed67ef952db" containerName="maas-api"
Oct 31 03:59:50.852962 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852446    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="de5b552a-3495-40e0-8f75-8fd08752474e" containerName="payload-processing"
Oct 31 03:59:50.852962 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852451    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="00e44046-fb2c-481f-8221-190358d77d1a" containerName="main"
Oct 31 03:59:50.852962 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852457    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="c29a47d5-b793-43c1-befc-b32b16e31114" containerName="istio-proxy"
Oct 31 03:59:50.852962 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852462    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f019db-a086-4f4b-b1cf-499c6aacc158" containerName="payload-pre-processing"
Oct 31 03:59:50.852962 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852468    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="b07daa8e-894c-42e3-ac95-a21c71b8735c" containerName="payload-processing"
Oct 31 03:59:50.852962 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.852472    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc56ecc-85d1-4a46-b567-8ece499dba21" containerName="revoke-keys"
Oct 31 03:59:50.857323 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.857303    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4418c521-p8kzs"
Oct 31 03:59:50.915735 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.915703    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4418c521-p8kzs"]
Oct 31 03:59:50.952642 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.952607    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b312195-5143-45c0-b3ac-5438944c007b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4418c521-p8kzs\" (UID: \"0b312195-5143-45c0-b3ac-5438944c007b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4418c521-p8kzs"
Oct 31 03:59:50.952844 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.952647    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl5f6\" (UniqueName: \"kubernetes.io/projected/0b312195-5143-45c0-b3ac-5438944c007b-kube-api-access-vl5f6\") pod \"maas-api-revoke-sub-4418c521-p8kzs\" (UID: \"0b312195-5143-45c0-b3ac-5438944c007b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4418c521-p8kzs"
Oct 31 03:59:50.952976 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.952944    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b312195-5143-45c0-b3ac-5438944c007b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4418c521-p8kzs\" (UID: \"0b312195-5143-45c0-b3ac-5438944c007b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4418c521-p8kzs"
Oct 31 03:59:50.961239 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.961210    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:59:50.961239 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:50.961235    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 03:59:51.053744 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:51.053713    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b312195-5143-45c0-b3ac-5438944c007b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4418c521-p8kzs\" (UID: \"0b312195-5143-45c0-b3ac-5438944c007b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4418c521-p8kzs"
Oct 31 03:59:51.053986 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:51.053850    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b312195-5143-45c0-b3ac-5438944c007b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4418c521-p8kzs\" (UID: \"0b312195-5143-45c0-b3ac-5438944c007b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4418c521-p8kzs"
Oct 31 03:59:51.053986 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:51.053876    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl5f6\" (UniqueName: \"kubernetes.io/projected/0b312195-5143-45c0-b3ac-5438944c007b-kube-api-access-vl5f6\") pod \"maas-api-revoke-sub-4418c521-p8kzs\" (UID: \"0b312195-5143-45c0-b3ac-5438944c007b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4418c521-p8kzs"
Oct 31 03:59:51.054498 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:51.054472    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b312195-5143-45c0-b3ac-5438944c007b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4418c521-p8kzs\" (UID: \"0b312195-5143-45c0-b3ac-5438944c007b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4418c521-p8kzs"
Oct 31 03:59:51.056312 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:51.056289    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b312195-5143-45c0-b3ac-5438944c007b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4418c521-p8kzs\" (UID: \"0b312195-5143-45c0-b3ac-5438944c007b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4418c521-p8kzs"
Oct 31 03:59:51.126934 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:51.126785    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl5f6\" (UniqueName: \"kubernetes.io/projected/0b312195-5143-45c0-b3ac-5438944c007b-kube-api-access-vl5f6\") pod \"maas-api-revoke-sub-4418c521-p8kzs\" (UID: \"0b312195-5143-45c0-b3ac-5438944c007b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4418c521-p8kzs"
Oct 31 03:59:51.172683 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:51.172651    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4418c521-p8kzs"
Oct 31 03:59:51.550765 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:51.550738    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4418c521-p8kzs"]
Oct 31 03:59:51.553224 ip-10-0-134-255 kubenswrapper[4447]: W1031 03:59:51.553199    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b312195_5143_45c0_b3ac_5438944c007b.slice/crio-2147a066a0b3b97ffadfa23fab5cbbdce0a7890f23c45e3652f99ea365a7ed85 WatchSource:0}: Error finding container 2147a066a0b3b97ffadfa23fab5cbbdce0a7890f23c45e3652f99ea365a7ed85: Status 404 returned error can't find the container with id 2147a066a0b3b97ffadfa23fab5cbbdce0a7890f23c45e3652f99ea365a7ed85
Oct 31 03:59:52.055984 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:52.055941    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4418c521-p8kzs" event={"ID":"0b312195-5143-45c0-b3ac-5438944c007b","Type":"ContainerStarted","Data":"d488d932039dd2ee9f5b8790a6a0192cac0fa4938f7c1746c8f91df2dad56685"}
Oct 31 03:59:52.056382 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:52.056009    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4418c521-p8kzs" event={"ID":"0b312195-5143-45c0-b3ac-5438944c007b","Type":"ContainerStarted","Data":"2147a066a0b3b97ffadfa23fab5cbbdce0a7890f23c45e3652f99ea365a7ed85"}
Oct 31 03:59:52.173222 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:52.173101    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4418c521-p8kzs" podStartSLOduration=2.173086727 podStartE2EDuration="2.173086727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:59:52.170936808 +0000 UTC m=+2417.681919186" watchObservedRunningTime="2026-10-31 03:59:52.173086727 +0000 UTC m=+2417.684069103"
Oct 31 03:59:53.068053 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:53.068016    4447 generic.go:356] "Generic (PLEG): container finished" podID="0b312195-5143-45c0-b3ac-5438944c007b" containerID="d488d932039dd2ee9f5b8790a6a0192cac0fa4938f7c1746c8f91df2dad56685" exitCode=0
Oct 31 03:59:53.068435 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:53.068093    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4418c521-p8kzs" event={"ID":"0b312195-5143-45c0-b3ac-5438944c007b","Type":"ContainerDied","Data":"d488d932039dd2ee9f5b8790a6a0192cac0fa4938f7c1746c8f91df2dad56685"}
Oct 31 03:59:54.224732 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:54.224707    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4418c521-p8kzs"
Oct 31 03:59:54.293475 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:54.293443    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b312195-5143-45c0-b3ac-5438944c007b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b312195-5143-45c0-b3ac-5438944c007b-maas-api-openshift-ca\") pod \"0b312195-5143-45c0-b3ac-5438944c007b\" (UID: \"0b312195-5143-45c0-b3ac-5438944c007b\") "
Oct 31 03:59:54.293656 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:54.293506    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b312195-5143-45c0-b3ac-5438944c007b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b312195-5143-45c0-b3ac-5438944c007b-maas-api-xks-ca\") pod \"0b312195-5143-45c0-b3ac-5438944c007b\" (UID: \"0b312195-5143-45c0-b3ac-5438944c007b\") "
Oct 31 03:59:54.293697 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:54.293671    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b312195-5143-45c0-b3ac-5438944c007b-kube-api-access-vl5f6\" (UniqueName: \"kubernetes.io/projected/0b312195-5143-45c0-b3ac-5438944c007b-kube-api-access-vl5f6\") pod \"0b312195-5143-45c0-b3ac-5438944c007b\" (UID: \"0b312195-5143-45c0-b3ac-5438944c007b\") "
Oct 31 03:59:54.293909 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:54.293867    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b312195-5143-45c0-b3ac-5438944c007b-maas-api-openshift-ca" pod "0b312195-5143-45c0-b3ac-5438944c007b" (UID: "0b312195-5143-45c0-b3ac-5438944c007b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:54.294065 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:54.294048    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b312195-5143-45c0-b3ac-5438944c007b-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:54.295932 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:54.295875    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b312195-5143-45c0-b3ac-5438944c007b-maas-api-xks-ca" pod "0b312195-5143-45c0-b3ac-5438944c007b" (UID: "0b312195-5143-45c0-b3ac-5438944c007b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:59:54.295932 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:54.295927    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b312195-5143-45c0-b3ac-5438944c007b-kube-api-access-vl5f6" pod "0b312195-5143-45c0-b3ac-5438944c007b" (UID: "0b312195-5143-45c0-b3ac-5438944c007b"). InnerVolumeSpecName "kube-api-access-vl5f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:54.395454 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:54.395368    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl5f6\" (UniqueName: \"kubernetes.io/projected/0b312195-5143-45c0-b3ac-5438944c007b-kube-api-access-vl5f6\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:54.395454 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:54.395397    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b312195-5143-45c0-b3ac-5438944c007b-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 03:59:54.530562 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:54.530516    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt" podUID="22c3bb42-e607-495f-942f-15072c8a0cf6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:59:55.084169 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:55.084136    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4418c521-p8kzs"
Oct 31 03:59:55.084169 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:55.084166    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4418c521-p8kzs" event={"ID":"0b312195-5143-45c0-b3ac-5438944c007b","Type":"ContainerDied","Data":"2147a066a0b3b97ffadfa23fab5cbbdce0a7890f23c45e3652f99ea365a7ed85"}
Oct 31 03:59:55.084397 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:55.084210    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2147a066a0b3b97ffadfa23fab5cbbdce0a7890f23c45e3652f99ea365a7ed85"
Oct 31 03:59:55.538182 ip-10-0-134-255 kubenswrapper[4447]: I1031 03:59:55.538142    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp" podUID="b6d42eac-1761-4ff4-a92c-9de57c7fb84a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:00:00.171801 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.171762    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d"]
Oct 31 04:00:00.172594 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.172571    4447 cpu_manager.go:414] "containerMap: removing container" podUID="0b312195-5143-45c0-b3ac-5438944c007b" containerName="revoke-keys"
Oct 31 04:00:00.172718 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.172597    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b312195-5143-45c0-b3ac-5438944c007b" containerName="revoke-keys"
Oct 31 04:00:00.172775 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.172749    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b312195-5143-45c0-b3ac-5438944c007b" containerName="revoke-keys"
Oct 31 04:00:00.176402 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.176377    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254"]
Oct 31 04:00:00.176538 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.176507    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d"
Oct 31 04:00:00.180432 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.180408    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254"
Oct 31 04:00:00.189916 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.189865    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx"]
Oct 31 04:00:00.194153 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.194136    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx"
Oct 31 04:00:00.228794 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.228758    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 04:00:00.270409 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.270363    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b44e13ed-4f1b-488c-8d16-89936835efaa-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890320-v5t8d\" (UID: \"b44e13ed-4f1b-488c-8d16-89936835efaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d"
Oct 31 04:00:00.270580 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.270542    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbs8h\" (UniqueName: \"kubernetes.io/projected/b44e13ed-4f1b-488c-8d16-89936835efaa-kube-api-access-dbs8h\") pod \"maas-api-key-cleanup-29890320-v5t8d\" (UID: \"b44e13ed-4f1b-488c-8d16-89936835efaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d"
Oct 31 04:00:00.297907 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.280036    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d"]
Oct 31 04:00:00.311670 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.311630    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254"]
Oct 31 04:00:00.371733 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.371693    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lj2s\" (UniqueName: \"kubernetes.io/projected/eabef4cb-bb40-46ef-8dfe-ce5e5df62d19-kube-api-access-9lj2s\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx\" (UID: \"eabef4cb-bb40-46ef-8dfe-ce5e5df62d19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx"
Oct 31 04:00:00.371981 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.371761    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbs8h\" (UniqueName: \"kubernetes.io/projected/b44e13ed-4f1b-488c-8d16-89936835efaa-kube-api-access-dbs8h\") pod \"maas-api-key-cleanup-29890320-v5t8d\" (UID: \"b44e13ed-4f1b-488c-8d16-89936835efaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d"
Oct 31 04:00:00.372073 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.372032    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9k9f\" (UniqueName: \"kubernetes.io/projected/e4b0b6eb-4592-47d4-ba16-afa891db7458-kube-api-access-v9k9f\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254\" (UID: \"e4b0b6eb-4592-47d4-ba16-afa891db7458\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254"
Oct 31 04:00:00.372173 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.372153    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabef4cb-bb40-46ef-8dfe-ce5e5df62d19-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx\" (UID: \"eabef4cb-bb40-46ef-8dfe-ce5e5df62d19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx"
Oct 31 04:00:00.372223 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.372197    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b44e13ed-4f1b-488c-8d16-89936835efaa-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890320-v5t8d\" (UID: \"b44e13ed-4f1b-488c-8d16-89936835efaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d"
Oct 31 04:00:00.372266 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.372232    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4b0b6eb-4592-47d4-ba16-afa891db7458-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254\" (UID: \"e4b0b6eb-4592-47d4-ba16-afa891db7458\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254"
Oct 31 04:00:00.373722 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.372979    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b44e13ed-4f1b-488c-8d16-89936835efaa-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890320-v5t8d\" (UID: \"b44e13ed-4f1b-488c-8d16-89936835efaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d"
Oct 31 04:00:00.375508 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.375472    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx"]
Oct 31 04:00:00.436393 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.436314    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbs8h\" (UniqueName: \"kubernetes.io/projected/b44e13ed-4f1b-488c-8d16-89936835efaa-kube-api-access-dbs8h\") pod \"maas-api-key-cleanup-29890320-v5t8d\" (UID: \"b44e13ed-4f1b-488c-8d16-89936835efaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d"
Oct 31 04:00:00.473148 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.473112    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9k9f\" (UniqueName: \"kubernetes.io/projected/e4b0b6eb-4592-47d4-ba16-afa891db7458-kube-api-access-v9k9f\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254\" (UID: \"e4b0b6eb-4592-47d4-ba16-afa891db7458\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254"
Oct 31 04:00:00.473308 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.473167    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabef4cb-bb40-46ef-8dfe-ce5e5df62d19-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx\" (UID: \"eabef4cb-bb40-46ef-8dfe-ce5e5df62d19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx"
Oct 31 04:00:00.473308 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.473192    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4b0b6eb-4592-47d4-ba16-afa891db7458-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254\" (UID: \"e4b0b6eb-4592-47d4-ba16-afa891db7458\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254"
Oct 31 04:00:00.473308 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.473257    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lj2s\" (UniqueName: \"kubernetes.io/projected/eabef4cb-bb40-46ef-8dfe-ce5e5df62d19-kube-api-access-9lj2s\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx\" (UID: \"eabef4cb-bb40-46ef-8dfe-ce5e5df62d19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx"
Oct 31 04:00:00.473863 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.473837    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabef4cb-bb40-46ef-8dfe-ce5e5df62d19-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx\" (UID: \"eabef4cb-bb40-46ef-8dfe-ce5e5df62d19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx"
Oct 31 04:00:00.474008 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.473844    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4b0b6eb-4592-47d4-ba16-afa891db7458-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254\" (UID: \"e4b0b6eb-4592-47d4-ba16-afa891db7458\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254"
Oct 31 04:00:00.492551 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.492517    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d"
Oct 31 04:00:00.534591 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.534556    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9k9f\" (UniqueName: \"kubernetes.io/projected/e4b0b6eb-4592-47d4-ba16-afa891db7458-kube-api-access-v9k9f\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254\" (UID: \"e4b0b6eb-4592-47d4-ba16-afa891db7458\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254"
Oct 31 04:00:00.534836 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.534814    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lj2s\" (UniqueName: \"kubernetes.io/projected/eabef4cb-bb40-46ef-8dfe-ce5e5df62d19-kube-api-access-9lj2s\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx\" (UID: \"eabef4cb-bb40-46ef-8dfe-ce5e5df62d19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx"
Oct 31 04:00:00.801911 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.798527    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254"
Oct 31 04:00:00.806350 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.806318    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx"
Oct 31 04:00:00.882057 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:00.882019    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d"]
Oct 31 04:00:01.127115 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:01.127084    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d" event={"ID":"b44e13ed-4f1b-488c-8d16-89936835efaa","Type":"ContainerStarted","Data":"ab74627810344b389bc0a7bd1f8a56b850a6e6f9fca21239b30a80edba18cd35"}
Oct 31 04:00:01.189942 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:01.189910    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254"]
Oct 31 04:00:01.190704 ip-10-0-134-255 kubenswrapper[4447]: W1031 04:00:01.190671    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b0b6eb_4592_47d4_ba16_afa891db7458.slice/crio-f5a71795f6b7b46141075ae35467ec5d0662bf17cc76bacccb772f2e950d45d6 WatchSource:0}: Error finding container f5a71795f6b7b46141075ae35467ec5d0662bf17cc76bacccb772f2e950d45d6: Status 404 returned error can't find the container with id f5a71795f6b7b46141075ae35467ec5d0662bf17cc76bacccb772f2e950d45d6
Oct 31 04:00:01.206879 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:01.206846    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx"]
Oct 31 04:00:01.208029 ip-10-0-134-255 kubenswrapper[4447]: W1031 04:00:01.207999    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeabef4cb_bb40_46ef_8dfe_ce5e5df62d19.slice/crio-29f8a7d6c8ce031a24e42678d2589b611900ea4d5a0de66b95a66872d0cb434b WatchSource:0}: Error finding container 29f8a7d6c8ce031a24e42678d2589b611900ea4d5a0de66b95a66872d0cb434b: Status 404 returned error can't find the container with id 29f8a7d6c8ce031a24e42678d2589b611900ea4d5a0de66b95a66872d0cb434b
Oct 31 04:00:02.135348 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:02.135312    4447 generic.go:356] "Generic (PLEG): container finished" podID="b44e13ed-4f1b-488c-8d16-89936835efaa" containerID="b7a7fd26f722441a5586b569734fa814e365f667271b920040699c3e523f61b0" exitCode=60
Oct 31 04:00:02.135499 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:02.135396    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d" event={"ID":"b44e13ed-4f1b-488c-8d16-89936835efaa","Type":"ContainerDied","Data":"b7a7fd26f722441a5586b569734fa814e365f667271b920040699c3e523f61b0"}
Oct 31 04:00:02.135751 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:02.135736    4447 scope.go:122] "RemoveContainer" containerID="b7a7fd26f722441a5586b569734fa814e365f667271b920040699c3e523f61b0"
Oct 31 04:00:02.137205 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:02.137185    4447 generic.go:356] "Generic (PLEG): container finished" podID="e4b0b6eb-4592-47d4-ba16-afa891db7458" containerID="f244c5f2fb24a7f6e1333d863cc54aae9f8980457219070d90c648c74d0e52f9" exitCode=60
Oct 31 04:00:02.137352 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:02.137291    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254" event={"ID":"e4b0b6eb-4592-47d4-ba16-afa891db7458","Type":"ContainerDied","Data":"f244c5f2fb24a7f6e1333d863cc54aae9f8980457219070d90c648c74d0e52f9"}
Oct 31 04:00:02.137352 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:02.137335    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254" event={"ID":"e4b0b6eb-4592-47d4-ba16-afa891db7458","Type":"ContainerStarted","Data":"f5a71795f6b7b46141075ae35467ec5d0662bf17cc76bacccb772f2e950d45d6"}
Oct 31 04:00:02.137609 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:02.137593    4447 scope.go:122] "RemoveContainer" containerID="f244c5f2fb24a7f6e1333d863cc54aae9f8980457219070d90c648c74d0e52f9"
Oct 31 04:00:02.139108 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:02.139051    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx" event={"ID":"eabef4cb-bb40-46ef-8dfe-ce5e5df62d19","Type":"ContainerStarted","Data":"c69f15ccd9199b9599a1dbcee0a9425288dff90cdd71e3308c93d9745b025fc4"}
Oct 31 04:00:02.139108 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:02.139094    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx" event={"ID":"eabef4cb-bb40-46ef-8dfe-ce5e5df62d19","Type":"ContainerStarted","Data":"29f8a7d6c8ce031a24e42678d2589b611900ea4d5a0de66b95a66872d0cb434b"}
Oct 31 04:00:02.487839 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:02.487782    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx" podStartSLOduration=2.487768504 podStartE2EDuration="2.487768504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:02.484432839 +0000 UTC m=+2427.995415217" watchObservedRunningTime="2026-10-31 04:00:02.487768504 +0000 UTC m=+2427.998750880"
Oct 31 04:00:03.148957 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:03.148916    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254" event={"ID":"e4b0b6eb-4592-47d4-ba16-afa891db7458","Type":"ContainerStarted","Data":"6bda748be0fc348a60c7215ac43f0aaefa2ce5c4173d07a92393184635d9bea3"}
Oct 31 04:00:03.151541 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:03.151507    4447 generic.go:356] "Generic (PLEG): container finished" podID="eabef4cb-bb40-46ef-8dfe-ce5e5df62d19" containerID="c69f15ccd9199b9599a1dbcee0a9425288dff90cdd71e3308c93d9745b025fc4" exitCode=60
Oct 31 04:00:03.151722 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:03.151667    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx" event={"ID":"eabef4cb-bb40-46ef-8dfe-ce5e5df62d19","Type":"ContainerDied","Data":"c69f15ccd9199b9599a1dbcee0a9425288dff90cdd71e3308c93d9745b025fc4"}
Oct 31 04:00:03.156057 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:03.152141    4447 scope.go:122] "RemoveContainer" containerID="c69f15ccd9199b9599a1dbcee0a9425288dff90cdd71e3308c93d9745b025fc4"
Oct 31 04:00:03.170335 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:03.170296    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d" event={"ID":"b44e13ed-4f1b-488c-8d16-89936835efaa","Type":"ContainerStarted","Data":"94de4c5d34801180948161f00cf85d389c112a5da1d99e0958ca9476a7b3fbd2"}
Oct 31 04:00:03.267249 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:03.267197    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254" podStartSLOduration=3.267182359 podStartE2EDuration="3.267182359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:03.264693537 +0000 UTC m=+2428.775675919" watchObservedRunningTime="2026-10-31 04:00:03.267182359 +0000 UTC m=+2428.778164780"
Oct 31 04:00:03.381293 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:03.381233    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d" podStartSLOduration=3.381215146 podStartE2EDuration="3.381215146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:03.381056228 +0000 UTC m=+2428.892038606" watchObservedRunningTime="2026-10-31 04:00:03.381215146 +0000 UTC m=+2428.892197525"
Oct 31 04:00:04.179215 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:04.179173    4447 generic.go:356] "Generic (PLEG): container finished" podID="b44e13ed-4f1b-488c-8d16-89936835efaa" containerID="94de4c5d34801180948161f00cf85d389c112a5da1d99e0958ca9476a7b3fbd2" exitCode=60
Oct 31 04:00:04.179635 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:04.179300    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d" event={"ID":"b44e13ed-4f1b-488c-8d16-89936835efaa","Type":"ContainerDied","Data":"94de4c5d34801180948161f00cf85d389c112a5da1d99e0958ca9476a7b3fbd2"}
Oct 31 04:00:04.179635 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:04.179368    4447 scope.go:122] "RemoveContainer" containerID="b7a7fd26f722441a5586b569734fa814e365f667271b920040699c3e523f61b0"
Oct 31 04:00:04.179635 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:04.179556    4447 scope.go:122] "RemoveContainer" containerID="94de4c5d34801180948161f00cf85d389c112a5da1d99e0958ca9476a7b3fbd2"
Oct 31 04:00:04.179948 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:00:04.179918    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890320-v5t8d_odh-ai-gateway-infra(b44e13ed-4f1b-488c-8d16-89936835efaa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d" podUID="b44e13ed-4f1b-488c-8d16-89936835efaa"
Oct 31 04:00:04.181357 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:04.181339    4447 generic.go:356] "Generic (PLEG): container finished" podID="e4b0b6eb-4592-47d4-ba16-afa891db7458" containerID="6bda748be0fc348a60c7215ac43f0aaefa2ce5c4173d07a92393184635d9bea3" exitCode=60
Oct 31 04:00:04.181477 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:04.181438    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254" event={"ID":"e4b0b6eb-4592-47d4-ba16-afa891db7458","Type":"ContainerDied","Data":"6bda748be0fc348a60c7215ac43f0aaefa2ce5c4173d07a92393184635d9bea3"}
Oct 31 04:00:04.181650 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:04.181635    4447 scope.go:122] "RemoveContainer" containerID="6bda748be0fc348a60c7215ac43f0aaefa2ce5c4173d07a92393184635d9bea3"
Oct 31 04:00:04.181955 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:00:04.181935    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254_odh-ai-gateway-infra(e4b0b6eb-4592-47d4-ba16-afa891db7458)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254" podUID="e4b0b6eb-4592-47d4-ba16-afa891db7458"
Oct 31 04:00:04.183375 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:04.183356    4447 generic.go:356] "Generic (PLEG): container finished" podID="eabef4cb-bb40-46ef-8dfe-ce5e5df62d19" containerID="162fb1ee61aa036c0130563fc39b9514a75233cb41f9871b2c8a399f4e7f618b" exitCode=60
Oct 31 04:00:04.183479 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:04.183442    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx" event={"ID":"eabef4cb-bb40-46ef-8dfe-ce5e5df62d19","Type":"ContainerDied","Data":"162fb1ee61aa036c0130563fc39b9514a75233cb41f9871b2c8a399f4e7f618b"}
Oct 31 04:00:04.183659 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:04.183643    4447 scope.go:122] "RemoveContainer" containerID="162fb1ee61aa036c0130563fc39b9514a75233cb41f9871b2c8a399f4e7f618b"
Oct 31 04:00:04.183916 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:00:04.183870    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx_odh-ai-gateway-infra(eabef4cb-bb40-46ef-8dfe-ce5e5df62d19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx" podUID="eabef4cb-bb40-46ef-8dfe-ce5e5df62d19"
Oct 31 04:00:04.192306 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:04.192287    4447 scope.go:122] "RemoveContainer" containerID="f244c5f2fb24a7f6e1333d863cc54aae9f8980457219070d90c648c74d0e52f9"
Oct 31 04:00:04.206115 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:04.206091    4447 scope.go:122] "RemoveContainer" containerID="c69f15ccd9199b9599a1dbcee0a9425288dff90cdd71e3308c93d9745b025fc4"
Oct 31 04:00:04.530243 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:04.530195    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt" podUID="22c3bb42-e607-495f-942f-15072c8a0cf6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:00:05.192025 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:05.192000    4447 scope.go:122] "RemoveContainer" containerID="6bda748be0fc348a60c7215ac43f0aaefa2ce5c4173d07a92393184635d9bea3"
Oct 31 04:00:05.192441 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:00:05.192321    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254_odh-ai-gateway-infra(e4b0b6eb-4592-47d4-ba16-afa891db7458)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254" podUID="e4b0b6eb-4592-47d4-ba16-afa891db7458"
Oct 31 04:00:05.193994 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:05.193969    4447 scope.go:122] "RemoveContainer" containerID="162fb1ee61aa036c0130563fc39b9514a75233cb41f9871b2c8a399f4e7f618b"
Oct 31 04:00:05.194237 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:00:05.194216    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx_odh-ai-gateway-infra(eabef4cb-bb40-46ef-8dfe-ce5e5df62d19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx" podUID="eabef4cb-bb40-46ef-8dfe-ce5e5df62d19"
Oct 31 04:00:05.195550 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:05.195536    4447 scope.go:122] "RemoveContainer" containerID="94de4c5d34801180948161f00cf85d389c112a5da1d99e0958ca9476a7b3fbd2"
Oct 31 04:00:05.195775 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:00:05.195758    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890320-v5t8d_odh-ai-gateway-infra(b44e13ed-4f1b-488c-8d16-89936835efaa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d" podUID="b44e13ed-4f1b-488c-8d16-89936835efaa"
Oct 31 04:00:05.538770 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:05.538730    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp" podUID="b6d42eac-1761-4ff4-a92c-9de57c7fb84a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:00:09.031001 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.030971    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp"
Oct 31 04:00:09.086742 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.086709    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d42eac-1761-4ff4-a92c-9de57c7fb84a-kube-api-access-67nns\" (UniqueName: \"kubernetes.io/projected/b6d42eac-1761-4ff4-a92c-9de57c7fb84a-kube-api-access-67nns\") pod \"b6d42eac-1761-4ff4-a92c-9de57c7fb84a\" (UID: \"b6d42eac-1761-4ff4-a92c-9de57c7fb84a\") "
Oct 31 04:00:09.087073 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.086985    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6d42eac-1761-4ff4-a92c-9de57c7fb84a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d42eac-1761-4ff4-a92c-9de57c7fb84a-plugins-config-volume\") pod \"b6d42eac-1761-4ff4-a92c-9de57c7fb84a\" (UID: \"b6d42eac-1761-4ff4-a92c-9de57c7fb84a\") "
Oct 31 04:00:09.087525 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.087489    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d42eac-1761-4ff4-a92c-9de57c7fb84a-plugins-config-volume" pod "b6d42eac-1761-4ff4-a92c-9de57c7fb84a" (UID: "b6d42eac-1761-4ff4-a92c-9de57c7fb84a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:09.089486 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.089460    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d42eac-1761-4ff4-a92c-9de57c7fb84a-kube-api-access-67nns" pod "b6d42eac-1761-4ff4-a92c-9de57c7fb84a" (UID: "b6d42eac-1761-4ff4-a92c-9de57c7fb84a"). InnerVolumeSpecName "kube-api-access-67nns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:09.116032 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.116010    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt"
Oct 31 04:00:09.188799 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.188771    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22c3bb42-e607-495f-942f-15072c8a0cf6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c3bb42-e607-495f-942f-15072c8a0cf6-plugins-config-volume\") pod \"22c3bb42-e607-495f-942f-15072c8a0cf6\" (UID: \"22c3bb42-e607-495f-942f-15072c8a0cf6\") "
Oct 31 04:00:09.189009 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.188849    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22c3bb42-e607-495f-942f-15072c8a0cf6-kube-api-access-nmn47\" (UniqueName: \"kubernetes.io/projected/22c3bb42-e607-495f-942f-15072c8a0cf6-kube-api-access-nmn47\") pod \"22c3bb42-e607-495f-942f-15072c8a0cf6\" (UID: \"22c3bb42-e607-495f-942f-15072c8a0cf6\") "
Oct 31 04:00:09.189172 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.189159    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d42eac-1761-4ff4-a92c-9de57c7fb84a-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:00:09.189209 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.189173    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67nns\" (UniqueName: \"kubernetes.io/projected/b6d42eac-1761-4ff4-a92c-9de57c7fb84a-kube-api-access-67nns\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:00:09.189240 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.189194    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c3bb42-e607-495f-942f-15072c8a0cf6-plugins-config-volume" pod "22c3bb42-e607-495f-942f-15072c8a0cf6" (UID: "22c3bb42-e607-495f-942f-15072c8a0cf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:09.191080 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.191058    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c3bb42-e607-495f-942f-15072c8a0cf6-kube-api-access-nmn47" pod "22c3bb42-e607-495f-942f-15072c8a0cf6" (UID: "22c3bb42-e607-495f-942f-15072c8a0cf6"). InnerVolumeSpecName "kube-api-access-nmn47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:09.225784 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.225750    4447 generic.go:356] "Generic (PLEG): container finished" podID="b6d42eac-1761-4ff4-a92c-9de57c7fb84a" containerID="113abeae90c9a322ae0507db8d9255c748f61c3a61a0f66ae09511f641eb27f1" exitCode=0
Oct 31 04:00:09.225956 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.225800    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp"
Oct 31 04:00:09.225956 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.225841    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp" event={"ID":"b6d42eac-1761-4ff4-a92c-9de57c7fb84a","Type":"ContainerDied","Data":"113abeae90c9a322ae0507db8d9255c748f61c3a61a0f66ae09511f641eb27f1"}
Oct 31 04:00:09.226058 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.225960    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp" event={"ID":"b6d42eac-1761-4ff4-a92c-9de57c7fb84a","Type":"ContainerDied","Data":"68c20c5da2f730f5de0a75e24d4de48cb45ef4af28c145a36cfa8da09fa71eef"}
Oct 31 04:00:09.226058 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.225989    4447 scope.go:122] "RemoveContainer" containerID="113abeae90c9a322ae0507db8d9255c748f61c3a61a0f66ae09511f641eb27f1"
Oct 31 04:00:09.227422 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.227401    4447 generic.go:356] "Generic (PLEG): container finished" podID="22c3bb42-e607-495f-942f-15072c8a0cf6" containerID="072c8bc8d8671e6930444bb66cb620e0ca4d3c38b8b9be366fc6962df2268eed" exitCode=0
Oct 31 04:00:09.227690 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.227436    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt"
Oct 31 04:00:09.227690 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.227472    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt" event={"ID":"22c3bb42-e607-495f-942f-15072c8a0cf6","Type":"ContainerDied","Data":"072c8bc8d8671e6930444bb66cb620e0ca4d3c38b8b9be366fc6962df2268eed"}
Oct 31 04:00:09.227690 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.227512    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt" event={"ID":"22c3bb42-e607-495f-942f-15072c8a0cf6","Type":"ContainerDied","Data":"387cf4fe7f925dd4f649be0e931f3de0df016bbb6afa0374cb7e9ee8b182e04e"}
Oct 31 04:00:09.237018 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.236982    4447 scope.go:122] "RemoveContainer" containerID="113abeae90c9a322ae0507db8d9255c748f61c3a61a0f66ae09511f641eb27f1"
Oct 31 04:00:09.237331 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:00:09.237305    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"113abeae90c9a322ae0507db8d9255c748f61c3a61a0f66ae09511f641eb27f1\": container with ID starting with 113abeae90c9a322ae0507db8d9255c748f61c3a61a0f66ae09511f641eb27f1 not found: ID does not exist" containerID="113abeae90c9a322ae0507db8d9255c748f61c3a61a0f66ae09511f641eb27f1"
Oct 31 04:00:09.237420 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.237342    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"113abeae90c9a322ae0507db8d9255c748f61c3a61a0f66ae09511f641eb27f1"} err="failed to get container status \"113abeae90c9a322ae0507db8d9255c748f61c3a61a0f66ae09511f641eb27f1\": rpc error: code = NotFound desc = could not find container \"113abeae90c9a322ae0507db8d9255c748f61c3a61a0f66ae09511f641eb27f1\": container with ID starting with 113abeae90c9a322ae0507db8d9255c748f61c3a61a0f66ae09511f641eb27f1 not found: ID does not exist"
Oct 31 04:00:09.237420 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.237359    4447 scope.go:122] "RemoveContainer" containerID="072c8bc8d8671e6930444bb66cb620e0ca4d3c38b8b9be366fc6962df2268eed"
Oct 31 04:00:09.267168 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.255312    4447 scope.go:122] "RemoveContainer" containerID="072c8bc8d8671e6930444bb66cb620e0ca4d3c38b8b9be366fc6962df2268eed"
Oct 31 04:00:09.267168 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:00:09.255771    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"072c8bc8d8671e6930444bb66cb620e0ca4d3c38b8b9be366fc6962df2268eed\": container with ID starting with 072c8bc8d8671e6930444bb66cb620e0ca4d3c38b8b9be366fc6962df2268eed not found: ID does not exist" containerID="072c8bc8d8671e6930444bb66cb620e0ca4d3c38b8b9be366fc6962df2268eed"
Oct 31 04:00:09.267168 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.255807    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"072c8bc8d8671e6930444bb66cb620e0ca4d3c38b8b9be366fc6962df2268eed"} err="failed to get container status \"072c8bc8d8671e6930444bb66cb620e0ca4d3c38b8b9be366fc6962df2268eed\": rpc error: code = NotFound desc = could not find container \"072c8bc8d8671e6930444bb66cb620e0ca4d3c38b8b9be366fc6962df2268eed\": container with ID starting with 072c8bc8d8671e6930444bb66cb620e0ca4d3c38b8b9be366fc6962df2268eed not found: ID does not exist"
Oct 31 04:00:09.289821 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.289789    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c3bb42-e607-495f-942f-15072c8a0cf6-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:00:09.289821 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.289818    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmn47\" (UniqueName: \"kubernetes.io/projected/22c3bb42-e607-495f-942f-15072c8a0cf6-kube-api-access-nmn47\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:00:09.419408 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.419374    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt"]
Oct 31 04:00:09.423920 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.423850    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-dec4a0-6554dd599f-qjtkt"]
Oct 31 04:00:09.533508 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.533423    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp"]
Oct 31 04:00:09.537717 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:09.537691    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-dec4a0-57c447f869-zkmzp"]
Oct 31 04:00:10.608479 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:10.608448    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22c3bb42-e607-495f-942f-15072c8a0cf6" path="/var/lib/kubelet/pods/22c3bb42-e607-495f-942f-15072c8a0cf6/volumes"
Oct 31 04:00:10.608840 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:10.608799    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6d42eac-1761-4ff4-a92c-9de57c7fb84a" path="/var/lib/kubelet/pods/b6d42eac-1761-4ff4-a92c-9de57c7fb84a/volumes"
Oct 31 04:00:16.600721 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:16.600686    4447 scope.go:122] "RemoveContainer" containerID="6bda748be0fc348a60c7215ac43f0aaefa2ce5c4173d07a92393184635d9bea3"
Oct 31 04:00:16.601408 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:16.600793    4447 scope.go:122] "RemoveContainer" containerID="94de4c5d34801180948161f00cf85d389c112a5da1d99e0958ca9476a7b3fbd2"
Oct 31 04:00:17.286798 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:17.286768    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d" event={"ID":"b44e13ed-4f1b-488c-8d16-89936835efaa","Type":"ContainerStarted","Data":"b7479223bf09fffe96825ef4ab23817a17d812df5805eb2dd422236650f300a7"}
Oct 31 04:00:17.288624 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:17.288601    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254" event={"ID":"e4b0b6eb-4592-47d4-ba16-afa891db7458","Type":"ContainerStarted","Data":"52aa61e743bf46313535e1f0ac1fb11516ac1181f83ebd346bae89e5e6c0a879"}
Oct 31 04:00:18.297460 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:18.297425    4447 generic.go:356] "Generic (PLEG): container finished" podID="e4b0b6eb-4592-47d4-ba16-afa891db7458" containerID="52aa61e743bf46313535e1f0ac1fb11516ac1181f83ebd346bae89e5e6c0a879" exitCode=60
Oct 31 04:00:18.297940 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:18.297497    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254" event={"ID":"e4b0b6eb-4592-47d4-ba16-afa891db7458","Type":"ContainerDied","Data":"52aa61e743bf46313535e1f0ac1fb11516ac1181f83ebd346bae89e5e6c0a879"}
Oct 31 04:00:18.297940 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:18.297557    4447 scope.go:122] "RemoveContainer" containerID="6bda748be0fc348a60c7215ac43f0aaefa2ce5c4173d07a92393184635d9bea3"
Oct 31 04:00:18.298094 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:18.297948    4447 scope.go:122] "RemoveContainer" containerID="52aa61e743bf46313535e1f0ac1fb11516ac1181f83ebd346bae89e5e6c0a879"
Oct 31 04:00:18.298295 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:00:18.298276    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254_odh-ai-gateway-infra(e4b0b6eb-4592-47d4-ba16-afa891db7458)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254" podUID="e4b0b6eb-4592-47d4-ba16-afa891db7458"
Oct 31 04:00:18.299750 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:18.299729    4447 generic.go:356] "Generic (PLEG): container finished" podID="b44e13ed-4f1b-488c-8d16-89936835efaa" containerID="b7479223bf09fffe96825ef4ab23817a17d812df5805eb2dd422236650f300a7" exitCode=60
Oct 31 04:00:18.299750 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:18.299759    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d" event={"ID":"b44e13ed-4f1b-488c-8d16-89936835efaa","Type":"ContainerDied","Data":"b7479223bf09fffe96825ef4ab23817a17d812df5805eb2dd422236650f300a7"}
Oct 31 04:00:18.300120 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:18.300108    4447 scope.go:122] "RemoveContainer" containerID="b7479223bf09fffe96825ef4ab23817a17d812df5805eb2dd422236650f300a7"
Oct 31 04:00:18.300359 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:00:18.300343    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890320-v5t8d_odh-ai-gateway-infra(b44e13ed-4f1b-488c-8d16-89936835efaa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d" podUID="b44e13ed-4f1b-488c-8d16-89936835efaa"
Oct 31 04:00:18.310357 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:18.310328    4447 scope.go:122] "RemoveContainer" containerID="94de4c5d34801180948161f00cf85d389c112a5da1d99e0958ca9476a7b3fbd2"
Oct 31 04:00:18.414047 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:18.414013    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d"]
Oct 31 04:00:18.529966 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:18.529932    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254"]
Oct 31 04:00:18.601048 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:18.600965    4447 scope.go:122] "RemoveContainer" containerID="162fb1ee61aa036c0130563fc39b9514a75233cb41f9871b2c8a399f4e7f618b"
Oct 31 04:00:19.003726 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:00:19.003684    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:00:19.309834 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:19.309801    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx" event={"ID":"eabef4cb-bb40-46ef-8dfe-ce5e5df62d19","Type":"ContainerStarted","Data":"936e28d66001e7cd08bbda3e054ab7ed02e038c4951c185ab362bae663db6f03"}
Oct 31 04:00:19.469480 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:19.469453    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254"
Oct 31 04:00:19.526775 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:19.526739    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4b0b6eb-4592-47d4-ba16-afa891db7458-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4b0b6eb-4592-47d4-ba16-afa891db7458-maas-api-service-ca\") pod \"e4b0b6eb-4592-47d4-ba16-afa891db7458\" (UID: \"e4b0b6eb-4592-47d4-ba16-afa891db7458\") "
Oct 31 04:00:19.526968 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:19.526869    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4b0b6eb-4592-47d4-ba16-afa891db7458-kube-api-access-v9k9f\" (UniqueName: \"kubernetes.io/projected/e4b0b6eb-4592-47d4-ba16-afa891db7458-kube-api-access-v9k9f\") pod \"e4b0b6eb-4592-47d4-ba16-afa891db7458\" (UID: \"e4b0b6eb-4592-47d4-ba16-afa891db7458\") "
Oct 31 04:00:19.527209 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:19.527184    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b0b6eb-4592-47d4-ba16-afa891db7458-maas-api-service-ca" pod "e4b0b6eb-4592-47d4-ba16-afa891db7458" (UID: "e4b0b6eb-4592-47d4-ba16-afa891db7458"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:19.527314 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:19.527301    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4b0b6eb-4592-47d4-ba16-afa891db7458-maas-api-service-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:00:19.529100 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:19.529080    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b0b6eb-4592-47d4-ba16-afa891db7458-kube-api-access-v9k9f" pod "e4b0b6eb-4592-47d4-ba16-afa891db7458" (UID: "e4b0b6eb-4592-47d4-ba16-afa891db7458"). InnerVolumeSpecName "kube-api-access-v9k9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:19.584555 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:19.584490    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d"
Oct 31 04:00:19.628559 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:19.628526    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b44e13ed-4f1b-488c-8d16-89936835efaa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b44e13ed-4f1b-488c-8d16-89936835efaa-maas-api-service-ca\") pod \"b44e13ed-4f1b-488c-8d16-89936835efaa\" (UID: \"b44e13ed-4f1b-488c-8d16-89936835efaa\") "
Oct 31 04:00:19.628765 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:19.628578    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b44e13ed-4f1b-488c-8d16-89936835efaa-kube-api-access-dbs8h\" (UniqueName: \"kubernetes.io/projected/b44e13ed-4f1b-488c-8d16-89936835efaa-kube-api-access-dbs8h\") pod \"b44e13ed-4f1b-488c-8d16-89936835efaa\" (UID: \"b44e13ed-4f1b-488c-8d16-89936835efaa\") "
Oct 31 04:00:19.628953 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:19.628931    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44e13ed-4f1b-488c-8d16-89936835efaa-maas-api-service-ca" pod "b44e13ed-4f1b-488c-8d16-89936835efaa" (UID: "b44e13ed-4f1b-488c-8d16-89936835efaa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:19.629049 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:19.629037    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9k9f\" (UniqueName: \"kubernetes.io/projected/e4b0b6eb-4592-47d4-ba16-afa891db7458-kube-api-access-v9k9f\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:00:19.629109 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:19.629084    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b44e13ed-4f1b-488c-8d16-89936835efaa-maas-api-service-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:00:19.630876 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:19.630853    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44e13ed-4f1b-488c-8d16-89936835efaa-kube-api-access-dbs8h" pod "b44e13ed-4f1b-488c-8d16-89936835efaa" (UID: "b44e13ed-4f1b-488c-8d16-89936835efaa"). InnerVolumeSpecName "kube-api-access-dbs8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:19.731048 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:19.730904    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbs8h\" (UniqueName: \"kubernetes.io/projected/b44e13ed-4f1b-488c-8d16-89936835efaa-kube-api-access-dbs8h\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:00:20.321209 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:20.321173    4447 generic.go:356] "Generic (PLEG): container finished" podID="eabef4cb-bb40-46ef-8dfe-ce5e5df62d19" containerID="936e28d66001e7cd08bbda3e054ab7ed02e038c4951c185ab362bae663db6f03" exitCode=60
Oct 31 04:00:20.321714 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:20.321244    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx" event={"ID":"eabef4cb-bb40-46ef-8dfe-ce5e5df62d19","Type":"ContainerDied","Data":"936e28d66001e7cd08bbda3e054ab7ed02e038c4951c185ab362bae663db6f03"}
Oct 31 04:00:20.321714 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:20.321308    4447 scope.go:122] "RemoveContainer" containerID="162fb1ee61aa036c0130563fc39b9514a75233cb41f9871b2c8a399f4e7f618b"
Oct 31 04:00:20.321714 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:20.321680    4447 scope.go:122] "RemoveContainer" containerID="936e28d66001e7cd08bbda3e054ab7ed02e038c4951c185ab362bae663db6f03"
Oct 31 04:00:20.322067 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:00:20.322041    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx_odh-ai-gateway-infra(eabef4cb-bb40-46ef-8dfe-ce5e5df62d19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx" podUID="eabef4cb-bb40-46ef-8dfe-ce5e5df62d19"
Oct 31 04:00:20.322970 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:20.322953    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d"
Oct 31 04:00:20.323086 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:20.322986    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d" event={"ID":"b44e13ed-4f1b-488c-8d16-89936835efaa","Type":"ContainerDied","Data":"ab74627810344b389bc0a7bd1f8a56b850a6e6f9fca21239b30a80edba18cd35"}
Oct 31 04:00:20.324485 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:20.324464    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254" event={"ID":"e4b0b6eb-4592-47d4-ba16-afa891db7458","Type":"ContainerDied","Data":"f5a71795f6b7b46141075ae35467ec5d0662bf17cc76bacccb772f2e950d45d6"}
Oct 31 04:00:20.324573 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:20.324475    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254"
Oct 31 04:00:20.335715 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:20.335695    4447 scope.go:122] "RemoveContainer" containerID="b7479223bf09fffe96825ef4ab23817a17d812df5805eb2dd422236650f300a7"
Oct 31 04:00:20.345508 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:20.345478    4447 scope.go:122] "RemoveContainer" containerID="52aa61e743bf46313535e1f0ac1fb11516ac1181f83ebd346bae89e5e6c0a879"
Oct 31 04:00:20.556936 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:20.556852    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx"]
Oct 31 04:00:20.561226 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:20.561038    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254"]
Oct 31 04:00:20.566777 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:20.566748    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-8de398-29890320-zs254"]
Oct 31 04:00:20.613774 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:20.613693    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4b0b6eb-4592-47d4-ba16-afa891db7458" path="/var/lib/kubelet/pods/e4b0b6eb-4592-47d4-ba16-afa891db7458/volumes"
Oct 31 04:00:20.679864 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:20.679823    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d"]
Oct 31 04:00:20.683609 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:20.683584    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890320-v5t8d"]
Oct 31 04:00:21.525057 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:21.525024    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx"
Oct 31 04:00:21.660326 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:21.660229    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eabef4cb-bb40-46ef-8dfe-ce5e5df62d19-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabef4cb-bb40-46ef-8dfe-ce5e5df62d19-maas-api-service-ca\") pod \"eabef4cb-bb40-46ef-8dfe-ce5e5df62d19\" (UID: \"eabef4cb-bb40-46ef-8dfe-ce5e5df62d19\") "
Oct 31 04:00:21.660500 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:21.660399    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eabef4cb-bb40-46ef-8dfe-ce5e5df62d19-kube-api-access-9lj2s\" (UniqueName: \"kubernetes.io/projected/eabef4cb-bb40-46ef-8dfe-ce5e5df62d19-kube-api-access-9lj2s\") pod \"eabef4cb-bb40-46ef-8dfe-ce5e5df62d19\" (UID: \"eabef4cb-bb40-46ef-8dfe-ce5e5df62d19\") "
Oct 31 04:00:21.661248 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:21.661216    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eabef4cb-bb40-46ef-8dfe-ce5e5df62d19-maas-api-service-ca" pod "eabef4cb-bb40-46ef-8dfe-ce5e5df62d19" (UID: "eabef4cb-bb40-46ef-8dfe-ce5e5df62d19"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:21.663197 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:21.663170    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eabef4cb-bb40-46ef-8dfe-ce5e5df62d19-kube-api-access-9lj2s" pod "eabef4cb-bb40-46ef-8dfe-ce5e5df62d19" (UID: "eabef4cb-bb40-46ef-8dfe-ce5e5df62d19"). InnerVolumeSpecName "kube-api-access-9lj2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:21.761898 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:21.761847    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lj2s\" (UniqueName: \"kubernetes.io/projected/eabef4cb-bb40-46ef-8dfe-ce5e5df62d19-kube-api-access-9lj2s\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:00:21.761898 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:21.761879    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabef4cb-bb40-46ef-8dfe-ce5e5df62d19-maas-api-service-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:00:22.356739 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:22.356714    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx"
Oct 31 04:00:22.356946 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:22.356707    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx" event={"ID":"eabef4cb-bb40-46ef-8dfe-ce5e5df62d19","Type":"ContainerDied","Data":"29f8a7d6c8ce031a24e42678d2589b611900ea4d5a0de66b95a66872d0cb434b"}
Oct 31 04:00:22.356946 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:22.356844    4447 scope.go:122] "RemoveContainer" containerID="936e28d66001e7cd08bbda3e054ab7ed02e038c4951c185ab362bae663db6f03"
Oct 31 04:00:22.521925 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:22.521874    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx"]
Oct 31 04:00:22.571492 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:22.571457    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-6de37a-29890320-tftxx"]
Oct 31 04:00:22.609091 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:22.609014    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b44e13ed-4f1b-488c-8d16-89936835efaa" path="/var/lib/kubelet/pods/b44e13ed-4f1b-488c-8d16-89936835efaa/volumes"
Oct 31 04:00:22.609480 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:22.609466    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eabef4cb-bb40-46ef-8dfe-ce5e5df62d19" path="/var/lib/kubelet/pods/eabef4cb-bb40-46ef-8dfe-ce5e5df62d19/volumes"
Oct 31 04:00:36.982959 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:36.982926    4447 scope.go:122] "RemoveContainer" containerID="e9c48ca3ef9116e1320fc9c0de398d17bcf70061bbe527d774f2ade99bb3aaf4"
Oct 31 04:00:36.992663 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:36.992645    4447 scope.go:122] "RemoveContainer" containerID="f43e12383984fdb0bb785e3f050af01be2fab1ecbd8ffe695afd6c21f4a94222"
Oct 31 04:00:49.002736 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:00:49.002657    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:00:49.596982 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:49.596949    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l"]
Oct 31 04:00:49.706694 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:49.706633    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp"]
Oct 31 04:00:54.597860 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:54.597798    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l" podUID="044e191f-f1a6-4e6a-89e5-50da9020417d" containerName="payload-processing" containerID="cri-o://29f63dce7e02bbd91cedcf7e82829100f8ca6164abc9efc999353887a350b8a9" gracePeriod=25
Oct 31 04:00:54.708109 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:54.708039    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp" podUID="ec8b3d1c-03b9-4c9a-8ef2-819411125551" containerName="payload-pre-processing" containerID="cri-o://2332b9aa66a169c7811163bc54fe7251b0d67187d78d1896eccfb8ac709cead9" gracePeriod=25
Oct 31 04:00:54.896307 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:54.896279    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l"
Oct 31 04:00:55.011643 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.011615    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp"
Oct 31 04:00:55.037805 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.037778    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/044e191f-f1a6-4e6a-89e5-50da9020417d-kube-api-access-jkv47\" (UniqueName: \"kubernetes.io/projected/044e191f-f1a6-4e6a-89e5-50da9020417d-kube-api-access-jkv47\") pod \"044e191f-f1a6-4e6a-89e5-50da9020417d\" (UID: \"044e191f-f1a6-4e6a-89e5-50da9020417d\") "
Oct 31 04:00:55.037991 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.037978    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/044e191f-f1a6-4e6a-89e5-50da9020417d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/044e191f-f1a6-4e6a-89e5-50da9020417d-plugins-config-volume\") pod \"044e191f-f1a6-4e6a-89e5-50da9020417d\" (UID: \"044e191f-f1a6-4e6a-89e5-50da9020417d\") "
Oct 31 04:00:55.038333 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.038309    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044e191f-f1a6-4e6a-89e5-50da9020417d-plugins-config-volume" pod "044e191f-f1a6-4e6a-89e5-50da9020417d" (UID: "044e191f-f1a6-4e6a-89e5-50da9020417d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:55.039963 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.039924    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044e191f-f1a6-4e6a-89e5-50da9020417d-kube-api-access-jkv47" pod "044e191f-f1a6-4e6a-89e5-50da9020417d" (UID: "044e191f-f1a6-4e6a-89e5-50da9020417d"). InnerVolumeSpecName "kube-api-access-jkv47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:55.146764 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.146678    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec8b3d1c-03b9-4c9a-8ef2-819411125551-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8b3d1c-03b9-4c9a-8ef2-819411125551-plugins-config-volume\") pod \"ec8b3d1c-03b9-4c9a-8ef2-819411125551\" (UID: \"ec8b3d1c-03b9-4c9a-8ef2-819411125551\") "
Oct 31 04:00:55.146976 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.146952    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec8b3d1c-03b9-4c9a-8ef2-819411125551-kube-api-access-hczd5\" (UniqueName: \"kubernetes.io/projected/ec8b3d1c-03b9-4c9a-8ef2-819411125551-kube-api-access-hczd5\") pod \"ec8b3d1c-03b9-4c9a-8ef2-819411125551\" (UID: \"ec8b3d1c-03b9-4c9a-8ef2-819411125551\") "
Oct 31 04:00:55.147444 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.147422    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/044e191f-f1a6-4e6a-89e5-50da9020417d-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:00:55.147542 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.147447    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkv47\" (UniqueName: \"kubernetes.io/projected/044e191f-f1a6-4e6a-89e5-50da9020417d-kube-api-access-jkv47\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:00:55.147609 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.147579    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec8b3d1c-03b9-4c9a-8ef2-819411125551-plugins-config-volume" pod "ec8b3d1c-03b9-4c9a-8ef2-819411125551" (UID: "ec8b3d1c-03b9-4c9a-8ef2-819411125551"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:55.149869 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.149846    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8b3d1c-03b9-4c9a-8ef2-819411125551-kube-api-access-hczd5" pod "ec8b3d1c-03b9-4c9a-8ef2-819411125551" (UID: "ec8b3d1c-03b9-4c9a-8ef2-819411125551"). InnerVolumeSpecName "kube-api-access-hczd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:55.248952 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.248910    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hczd5\" (UniqueName: \"kubernetes.io/projected/ec8b3d1c-03b9-4c9a-8ef2-819411125551-kube-api-access-hczd5\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:00:55.249194 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.249166    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8b3d1c-03b9-4c9a-8ef2-819411125551-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:00:55.626677 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.626639    4447 generic.go:356] "Generic (PLEG): container finished" podID="ec8b3d1c-03b9-4c9a-8ef2-819411125551" containerID="2332b9aa66a169c7811163bc54fe7251b0d67187d78d1896eccfb8ac709cead9" exitCode=0
Oct 31 04:00:55.627168 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.626724    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp"
Oct 31 04:00:55.627168 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.626721    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp" event={"ID":"ec8b3d1c-03b9-4c9a-8ef2-819411125551","Type":"ContainerDied","Data":"2332b9aa66a169c7811163bc54fe7251b0d67187d78d1896eccfb8ac709cead9"}
Oct 31 04:00:55.627168 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.626864    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp" event={"ID":"ec8b3d1c-03b9-4c9a-8ef2-819411125551","Type":"ContainerDied","Data":"31e161dd287335115191ddbec079419591940a1a5cc04283654abbba667d6abb"}
Oct 31 04:00:55.627168 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.626915    4447 scope.go:122] "RemoveContainer" containerID="2332b9aa66a169c7811163bc54fe7251b0d67187d78d1896eccfb8ac709cead9"
Oct 31 04:00:55.628374 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.628353    4447 generic.go:356] "Generic (PLEG): container finished" podID="044e191f-f1a6-4e6a-89e5-50da9020417d" containerID="29f63dce7e02bbd91cedcf7e82829100f8ca6164abc9efc999353887a350b8a9" exitCode=0
Oct 31 04:00:55.628465 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.628385    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l" event={"ID":"044e191f-f1a6-4e6a-89e5-50da9020417d","Type":"ContainerDied","Data":"29f63dce7e02bbd91cedcf7e82829100f8ca6164abc9efc999353887a350b8a9"}
Oct 31 04:00:55.628465 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.628416    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l" event={"ID":"044e191f-f1a6-4e6a-89e5-50da9020417d","Type":"ContainerDied","Data":"7f593627dc53512c67f340b84529ad74147289b7061e61a3365367837a2a1e85"}
Oct 31 04:00:55.628465 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.628421    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l"
Oct 31 04:00:55.639113 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.639089    4447 scope.go:122] "RemoveContainer" containerID="2332b9aa66a169c7811163bc54fe7251b0d67187d78d1896eccfb8ac709cead9"
Oct 31 04:00:55.639433 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:00:55.639405    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2332b9aa66a169c7811163bc54fe7251b0d67187d78d1896eccfb8ac709cead9\": container with ID starting with 2332b9aa66a169c7811163bc54fe7251b0d67187d78d1896eccfb8ac709cead9 not found: ID does not exist" containerID="2332b9aa66a169c7811163bc54fe7251b0d67187d78d1896eccfb8ac709cead9"
Oct 31 04:00:55.639509 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.639444    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2332b9aa66a169c7811163bc54fe7251b0d67187d78d1896eccfb8ac709cead9"} err="failed to get container status \"2332b9aa66a169c7811163bc54fe7251b0d67187d78d1896eccfb8ac709cead9\": rpc error: code = NotFound desc = could not find container \"2332b9aa66a169c7811163bc54fe7251b0d67187d78d1896eccfb8ac709cead9\": container with ID starting with 2332b9aa66a169c7811163bc54fe7251b0d67187d78d1896eccfb8ac709cead9 not found: ID does not exist"
Oct 31 04:00:55.639509 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.639462    4447 scope.go:122] "RemoveContainer" containerID="29f63dce7e02bbd91cedcf7e82829100f8ca6164abc9efc999353887a350b8a9"
Oct 31 04:00:55.648541 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.648524    4447 scope.go:122] "RemoveContainer" containerID="29f63dce7e02bbd91cedcf7e82829100f8ca6164abc9efc999353887a350b8a9"
Oct 31 04:00:55.648805 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:00:55.648783    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f63dce7e02bbd91cedcf7e82829100f8ca6164abc9efc999353887a350b8a9\": container with ID starting with 29f63dce7e02bbd91cedcf7e82829100f8ca6164abc9efc999353887a350b8a9 not found: ID does not exist" containerID="29f63dce7e02bbd91cedcf7e82829100f8ca6164abc9efc999353887a350b8a9"
Oct 31 04:00:55.648872 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.648817    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f63dce7e02bbd91cedcf7e82829100f8ca6164abc9efc999353887a350b8a9"} err="failed to get container status \"29f63dce7e02bbd91cedcf7e82829100f8ca6164abc9efc999353887a350b8a9\": rpc error: code = NotFound desc = could not find container \"29f63dce7e02bbd91cedcf7e82829100f8ca6164abc9efc999353887a350b8a9\": container with ID starting with 29f63dce7e02bbd91cedcf7e82829100f8ca6164abc9efc999353887a350b8a9 not found: ID does not exist"
Oct 31 04:00:55.793389 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.793353    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l"]
Oct 31 04:00:55.843017 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.842985    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-6de37a-7f64588c67-92w9l"]
Oct 31 04:00:55.908104 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.908012    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp"]
Oct 31 04:00:55.911374 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:55.911351    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-6de37a-75888bff59-crsmp"]
Oct 31 04:00:56.608795 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:56.608758    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="044e191f-f1a6-4e6a-89e5-50da9020417d" path="/var/lib/kubelet/pods/044e191f-f1a6-4e6a-89e5-50da9020417d/volumes"
Oct 31 04:00:56.609224 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:00:56.609209    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec8b3d1c-03b9-4c9a-8ef2-819411125551" path="/var/lib/kubelet/pods/ec8b3d1c-03b9-4c9a-8ef2-819411125551/volumes"
Oct 31 04:01:16.213676 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:16.213642    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z"]
Oct 31 04:01:16.321238 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:16.321202    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd"]
Oct 31 04:01:19.003266 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:01:19.003228    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:01:21.213977 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.213917    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z" podUID="58a17f12-6a7b-4645-b519-ea369877724d" containerName="payload-processing" containerID="cri-o://d2dd0312e5dfb336b619de12a80823e2aad780ea74ac5e8fa82cb13d54f0ccbb" gracePeriod=25
Oct 31 04:01:21.321870 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.321809    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd" podUID="994071b8-58c4-4431-8154-c00e67c7b065" containerName="payload-pre-processing" containerID="cri-o://b48f407747e1ac6b6eb828b59e4a29dc9e38e6661a6d2608b11d88d399229c99" gracePeriod=25
Oct 31 04:01:21.497072 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.497041    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z"
Oct 31 04:01:21.565687 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.565649    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a17f12-6a7b-4645-b519-ea369877724d-kube-api-access-kh6jz\" (UniqueName: \"kubernetes.io/projected/58a17f12-6a7b-4645-b519-ea369877724d-kube-api-access-kh6jz\") pod \"58a17f12-6a7b-4645-b519-ea369877724d\" (UID: \"58a17f12-6a7b-4645-b519-ea369877724d\") "
Oct 31 04:01:21.566087 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.566061    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58a17f12-6a7b-4645-b519-ea369877724d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a17f12-6a7b-4645-b519-ea369877724d-plugins-config-volume\") pod \"58a17f12-6a7b-4645-b519-ea369877724d\" (UID: \"58a17f12-6a7b-4645-b519-ea369877724d\") "
Oct 31 04:01:21.566950 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.566921    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a17f12-6a7b-4645-b519-ea369877724d-plugins-config-volume" pod "58a17f12-6a7b-4645-b519-ea369877724d" (UID: "58a17f12-6a7b-4645-b519-ea369877724d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:01:21.569205 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.569176    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a17f12-6a7b-4645-b519-ea369877724d-kube-api-access-kh6jz" pod "58a17f12-6a7b-4645-b519-ea369877724d" (UID: "58a17f12-6a7b-4645-b519-ea369877724d"). InnerVolumeSpecName "kube-api-access-kh6jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:21.614947 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.614921    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd"
Oct 31 04:01:21.667973 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.667941    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/994071b8-58c4-4431-8154-c00e67c7b065-kube-api-access-n2rd8\" (UniqueName: \"kubernetes.io/projected/994071b8-58c4-4431-8154-c00e67c7b065-kube-api-access-n2rd8\") pod \"994071b8-58c4-4431-8154-c00e67c7b065\" (UID: \"994071b8-58c4-4431-8154-c00e67c7b065\") "
Oct 31 04:01:21.668143 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.668036    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/994071b8-58c4-4431-8154-c00e67c7b065-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/994071b8-58c4-4431-8154-c00e67c7b065-plugins-config-volume\") pod \"994071b8-58c4-4431-8154-c00e67c7b065\" (UID: \"994071b8-58c4-4431-8154-c00e67c7b065\") "
Oct 31 04:01:21.668456 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.668428    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/994071b8-58c4-4431-8154-c00e67c7b065-plugins-config-volume" pod "994071b8-58c4-4431-8154-c00e67c7b065" (UID: "994071b8-58c4-4431-8154-c00e67c7b065"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:01:21.668583 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.668512    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh6jz\" (UniqueName: \"kubernetes.io/projected/58a17f12-6a7b-4645-b519-ea369877724d-kube-api-access-kh6jz\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:01:21.668583 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.668540    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/994071b8-58c4-4431-8154-c00e67c7b065-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:01:21.668583 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.668554    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a17f12-6a7b-4645-b519-ea369877724d-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:01:21.670325 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.670301    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994071b8-58c4-4431-8154-c00e67c7b065-kube-api-access-n2rd8" pod "994071b8-58c4-4431-8154-c00e67c7b065" (UID: "994071b8-58c4-4431-8154-c00e67c7b065"). InnerVolumeSpecName "kube-api-access-n2rd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:21.769935 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.769821    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2rd8\" (UniqueName: \"kubernetes.io/projected/994071b8-58c4-4431-8154-c00e67c7b065-kube-api-access-n2rd8\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:01:21.800333 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.800300    4447 generic.go:356] "Generic (PLEG): container finished" podID="994071b8-58c4-4431-8154-c00e67c7b065" containerID="b48f407747e1ac6b6eb828b59e4a29dc9e38e6661a6d2608b11d88d399229c99" exitCode=0
Oct 31 04:01:21.800529 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.800356    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd"
Oct 31 04:01:21.800529 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.800385    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd" event={"ID":"994071b8-58c4-4431-8154-c00e67c7b065","Type":"ContainerDied","Data":"b48f407747e1ac6b6eb828b59e4a29dc9e38e6661a6d2608b11d88d399229c99"}
Oct 31 04:01:21.800529 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.800458    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd" event={"ID":"994071b8-58c4-4431-8154-c00e67c7b065","Type":"ContainerDied","Data":"fecba42a1d5072c65154e9c684f8324e6779a15de29d3edda03cd98cbc07fc05"}
Oct 31 04:01:21.800529 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.800488    4447 scope.go:122] "RemoveContainer" containerID="b48f407747e1ac6b6eb828b59e4a29dc9e38e6661a6d2608b11d88d399229c99"
Oct 31 04:01:21.802138 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.802111    4447 generic.go:356] "Generic (PLEG): container finished" podID="58a17f12-6a7b-4645-b519-ea369877724d" containerID="d2dd0312e5dfb336b619de12a80823e2aad780ea74ac5e8fa82cb13d54f0ccbb" exitCode=0
Oct 31 04:01:21.802249 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.802237    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z"
Oct 31 04:01:21.802326 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.802238    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z" event={"ID":"58a17f12-6a7b-4645-b519-ea369877724d","Type":"ContainerDied","Data":"d2dd0312e5dfb336b619de12a80823e2aad780ea74ac5e8fa82cb13d54f0ccbb"}
Oct 31 04:01:21.802365 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.802355    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z" event={"ID":"58a17f12-6a7b-4645-b519-ea369877724d","Type":"ContainerDied","Data":"644be4b28c4a695ba3e495a3cc3710128297e6aa23ba97ca9c13d9a76e9eb61a"}
Oct 31 04:01:21.812235 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.812220    4447 scope.go:122] "RemoveContainer" containerID="b48f407747e1ac6b6eb828b59e4a29dc9e38e6661a6d2608b11d88d399229c99"
Oct 31 04:01:21.812478 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:01:21.812460    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b48f407747e1ac6b6eb828b59e4a29dc9e38e6661a6d2608b11d88d399229c99\": container with ID starting with b48f407747e1ac6b6eb828b59e4a29dc9e38e6661a6d2608b11d88d399229c99 not found: ID does not exist" containerID="b48f407747e1ac6b6eb828b59e4a29dc9e38e6661a6d2608b11d88d399229c99"
Oct 31 04:01:21.812531 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.812486    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b48f407747e1ac6b6eb828b59e4a29dc9e38e6661a6d2608b11d88d399229c99"} err="failed to get container status \"b48f407747e1ac6b6eb828b59e4a29dc9e38e6661a6d2608b11d88d399229c99\": rpc error: code = NotFound desc = could not find container \"b48f407747e1ac6b6eb828b59e4a29dc9e38e6661a6d2608b11d88d399229c99\": container with ID starting with b48f407747e1ac6b6eb828b59e4a29dc9e38e6661a6d2608b11d88d399229c99 not found: ID does not exist"
Oct 31 04:01:21.812531 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.812507    4447 scope.go:122] "RemoveContainer" containerID="d2dd0312e5dfb336b619de12a80823e2aad780ea74ac5e8fa82cb13d54f0ccbb"
Oct 31 04:01:21.821617 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.821601    4447 scope.go:122] "RemoveContainer" containerID="d2dd0312e5dfb336b619de12a80823e2aad780ea74ac5e8fa82cb13d54f0ccbb"
Oct 31 04:01:21.821855 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:01:21.821839    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2dd0312e5dfb336b619de12a80823e2aad780ea74ac5e8fa82cb13d54f0ccbb\": container with ID starting with d2dd0312e5dfb336b619de12a80823e2aad780ea74ac5e8fa82cb13d54f0ccbb not found: ID does not exist" containerID="d2dd0312e5dfb336b619de12a80823e2aad780ea74ac5e8fa82cb13d54f0ccbb"
Oct 31 04:01:21.821928 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.821862    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2dd0312e5dfb336b619de12a80823e2aad780ea74ac5e8fa82cb13d54f0ccbb"} err="failed to get container status \"d2dd0312e5dfb336b619de12a80823e2aad780ea74ac5e8fa82cb13d54f0ccbb\": rpc error: code = NotFound desc = could not find container \"d2dd0312e5dfb336b619de12a80823e2aad780ea74ac5e8fa82cb13d54f0ccbb\": container with ID starting with d2dd0312e5dfb336b619de12a80823e2aad780ea74ac5e8fa82cb13d54f0ccbb not found: ID does not exist"
Oct 31 04:01:21.966521 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.966489    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd"]
Oct 31 04:01:21.972909 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:21.972857    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-8de398-7b65c96c6c-srzqd"]
Oct 31 04:01:22.082377 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:22.082294    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z"]
Oct 31 04:01:22.086998 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:22.086972    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-8de398-579764c9c5-hj88z"]
Oct 31 04:01:22.609309 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:22.609277    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58a17f12-6a7b-4645-b519-ea369877724d" path="/var/lib/kubelet/pods/58a17f12-6a7b-4645-b519-ea369877724d/volumes"
Oct 31 04:01:22.609707 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:22.609693    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="994071b8-58c4-4431-8154-c00e67c7b065" path="/var/lib/kubelet/pods/994071b8-58c4-4431-8154-c00e67c7b065/volumes"
Oct 31 04:01:25.666555 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:25.666518    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"]
Oct 31 04:01:25.666980 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:25.666719    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r" podUID="be8c55fa-3e54-42e4-86f6-fe5f702b5df9" containerName="maas-api" containerID="cri-o://0cee3a7fa4e22c691df0679156bd95939c103be13ebe5bef29442d5107ce48fe" gracePeriod=30
Oct 31 04:01:25.833968 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:25.833927    4447 generic.go:356] "Generic (PLEG): container finished" podID="be8c55fa-3e54-42e4-86f6-fe5f702b5df9" containerID="0cee3a7fa4e22c691df0679156bd95939c103be13ebe5bef29442d5107ce48fe" exitCode=0
Oct 31 04:01:25.834133 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:25.833974    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r" event={"ID":"be8c55fa-3e54-42e4-86f6-fe5f702b5df9","Type":"ContainerDied","Data":"0cee3a7fa4e22c691df0679156bd95939c103be13ebe5bef29442d5107ce48fe"}
Oct 31 04:01:25.947082 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:25.947057    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"
Oct 31 04:01:26.016799 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:26.016763    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-metrics-certs\") pod \"be8c55fa-3e54-42e4-86f6-fe5f702b5df9\" (UID: \"be8c55fa-3e54-42e4-86f6-fe5f702b5df9\") "
Oct 31 04:01:26.017012 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:26.016857    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-openshift-service-ca\") pod \"be8c55fa-3e54-42e4-86f6-fe5f702b5df9\" (UID: \"be8c55fa-3e54-42e4-86f6-fe5f702b5df9\") "
Oct 31 04:01:26.017012 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:26.017004    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-maas-api-tls\") pod \"be8c55fa-3e54-42e4-86f6-fe5f702b5df9\" (UID: \"be8c55fa-3e54-42e4-86f6-fe5f702b5df9\") "
Oct 31 04:01:26.017091 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:26.017030    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-kube-api-access-ssq7f\" (UniqueName: \"kubernetes.io/projected/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-kube-api-access-ssq7f\") pod \"be8c55fa-3e54-42e4-86f6-fe5f702b5df9\" (UID: \"be8c55fa-3e54-42e4-86f6-fe5f702b5df9\") "
Oct 31 04:01:26.017242 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:26.017219    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-openshift-service-ca" pod "be8c55fa-3e54-42e4-86f6-fe5f702b5df9" (UID: "be8c55fa-3e54-42e4-86f6-fe5f702b5df9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:01:26.017445 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:26.017424    4447 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-openshift-service-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:01:26.019239 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:26.019209    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-metrics-certs" pod "be8c55fa-3e54-42e4-86f6-fe5f702b5df9" (UID: "be8c55fa-3e54-42e4-86f6-fe5f702b5df9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:01:26.019338 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:26.019248    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-maas-api-tls" pod "be8c55fa-3e54-42e4-86f6-fe5f702b5df9" (UID: "be8c55fa-3e54-42e4-86f6-fe5f702b5df9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:01:26.019338 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:26.019276    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-kube-api-access-ssq7f" pod "be8c55fa-3e54-42e4-86f6-fe5f702b5df9" (UID: "be8c55fa-3e54-42e4-86f6-fe5f702b5df9"). InnerVolumeSpecName "kube-api-access-ssq7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:26.118480 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:26.118449    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-maas-api-tls\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:01:26.118480 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:26.118479    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssq7f\" (UniqueName: \"kubernetes.io/projected/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-kube-api-access-ssq7f\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:01:26.118694 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:26.118490    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be8c55fa-3e54-42e4-86f6-fe5f702b5df9-metrics-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:01:26.842363 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:26.842327    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r" event={"ID":"be8c55fa-3e54-42e4-86f6-fe5f702b5df9","Type":"ContainerDied","Data":"835b915911159786501a931796efbd1b2890acdfc1609868c92b8875a93168a5"}
Oct 31 04:01:26.842363 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:26.842366    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"
Oct 31 04:01:26.842952 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:26.842387    4447 scope.go:122] "RemoveContainer" containerID="0cee3a7fa4e22c691df0679156bd95939c103be13ebe5bef29442d5107ce48fe"
Oct 31 04:01:27.009177 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:27.009137    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"]
Oct 31 04:01:27.013032 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:27.013008    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-8de398-78667c44b9-t9q8r"]
Oct 31 04:01:28.609036 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:28.609004    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be8c55fa-3e54-42e4-86f6-fe5f702b5df9" path="/var/lib/kubelet/pods/be8c55fa-3e54-42e4-86f6-fe5f702b5df9/volumes"
Oct 31 04:01:35.617992 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.617959    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg"]
Oct 31 04:01:35.618499 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618486    4447 cpu_manager.go:414] "containerMap: removing container" podUID="e4b0b6eb-4592-47d4-ba16-afa891db7458" containerName="cleanup"
Oct 31 04:01:35.618555 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618500    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4b0b6eb-4592-47d4-ba16-afa891db7458" containerName="cleanup"
Oct 31 04:01:35.618555 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618512    4447 cpu_manager.go:414] "containerMap: removing container" podUID="e4b0b6eb-4592-47d4-ba16-afa891db7458" containerName="cleanup"
Oct 31 04:01:35.618555 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618518    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4b0b6eb-4592-47d4-ba16-afa891db7458" containerName="cleanup"
Oct 31 04:01:35.618555 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618532    4447 cpu_manager.go:414] "containerMap: removing container" podUID="eabef4cb-bb40-46ef-8dfe-ce5e5df62d19" containerName="cleanup"
Oct 31 04:01:35.618555 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618537    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eabef4cb-bb40-46ef-8dfe-ce5e5df62d19" containerName="cleanup"
Oct 31 04:01:35.618555 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618545    4447 cpu_manager.go:414] "containerMap: removing container" podUID="eabef4cb-bb40-46ef-8dfe-ce5e5df62d19" containerName="cleanup"
Oct 31 04:01:35.618555 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618550    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eabef4cb-bb40-46ef-8dfe-ce5e5df62d19" containerName="cleanup"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618562    4447 cpu_manager.go:414] "containerMap: removing container" podUID="eabef4cb-bb40-46ef-8dfe-ce5e5df62d19" containerName="cleanup"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618566    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eabef4cb-bb40-46ef-8dfe-ce5e5df62d19" containerName="cleanup"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618574    4447 cpu_manager.go:414] "containerMap: removing container" podUID="e4b0b6eb-4592-47d4-ba16-afa891db7458" containerName="cleanup"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618579    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4b0b6eb-4592-47d4-ba16-afa891db7458" containerName="cleanup"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618586    4447 cpu_manager.go:414] "containerMap: removing container" podUID="b44e13ed-4f1b-488c-8d16-89936835efaa" containerName="cleanup"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618591    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44e13ed-4f1b-488c-8d16-89936835efaa" containerName="cleanup"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618596    4447 cpu_manager.go:414] "containerMap: removing container" podUID="044e191f-f1a6-4e6a-89e5-50da9020417d" containerName="payload-processing"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618602    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="044e191f-f1a6-4e6a-89e5-50da9020417d" containerName="payload-processing"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618609    4447 cpu_manager.go:414] "containerMap: removing container" podUID="b44e13ed-4f1b-488c-8d16-89936835efaa" containerName="cleanup"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618614    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44e13ed-4f1b-488c-8d16-89936835efaa" containerName="cleanup"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618626    4447 cpu_manager.go:414] "containerMap: removing container" podUID="b6d42eac-1761-4ff4-a92c-9de57c7fb84a" containerName="payload-processing"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618631    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d42eac-1761-4ff4-a92c-9de57c7fb84a" containerName="payload-processing"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618637    4447 cpu_manager.go:414] "containerMap: removing container" podUID="be8c55fa-3e54-42e4-86f6-fe5f702b5df9" containerName="maas-api"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618641    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be8c55fa-3e54-42e4-86f6-fe5f702b5df9" containerName="maas-api"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618651    4447 cpu_manager.go:414] "containerMap: removing container" podUID="ec8b3d1c-03b9-4c9a-8ef2-819411125551" containerName="payload-pre-processing"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618655    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec8b3d1c-03b9-4c9a-8ef2-819411125551" containerName="payload-pre-processing"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618662    4447 cpu_manager.go:414] "containerMap: removing container" podUID="58a17f12-6a7b-4645-b519-ea369877724d" containerName="payload-processing"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618667    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a17f12-6a7b-4645-b519-ea369877724d" containerName="payload-processing"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618675    4447 cpu_manager.go:414] "containerMap: removing container" podUID="b44e13ed-4f1b-488c-8d16-89936835efaa" containerName="cleanup"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618679    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44e13ed-4f1b-488c-8d16-89936835efaa" containerName="cleanup"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618687    4447 cpu_manager.go:414] "containerMap: removing container" podUID="994071b8-58c4-4431-8154-c00e67c7b065" containerName="payload-pre-processing"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618691    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="994071b8-58c4-4431-8154-c00e67c7b065" containerName="payload-pre-processing"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618699    4447 cpu_manager.go:414] "containerMap: removing container" podUID="22c3bb42-e607-495f-942f-15072c8a0cf6" containerName="payload-pre-processing"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618703    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22c3bb42-e607-495f-942f-15072c8a0cf6" containerName="payload-pre-processing"
Oct 31 04:01:35.618763 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618769    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d42eac-1761-4ff4-a92c-9de57c7fb84a" containerName="payload-processing"
Oct 31 04:01:35.619500 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618775    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="be8c55fa-3e54-42e4-86f6-fe5f702b5df9" containerName="maas-api"
Oct 31 04:01:35.619500 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618781    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec8b3d1c-03b9-4c9a-8ef2-819411125551" containerName="payload-pre-processing"
Oct 31 04:01:35.619500 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618788    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a17f12-6a7b-4645-b519-ea369877724d" containerName="payload-processing"
Oct 31 04:01:35.619500 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618793    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44e13ed-4f1b-488c-8d16-89936835efaa" containerName="cleanup"
Oct 31 04:01:35.619500 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618798    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="994071b8-58c4-4431-8154-c00e67c7b065" containerName="payload-pre-processing"
Oct 31 04:01:35.619500 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618805    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="22c3bb42-e607-495f-942f-15072c8a0cf6" containerName="payload-pre-processing"
Oct 31 04:01:35.619500 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618810    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4b0b6eb-4592-47d4-ba16-afa891db7458" containerName="cleanup"
Oct 31 04:01:35.619500 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618814    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4b0b6eb-4592-47d4-ba16-afa891db7458" containerName="cleanup"
Oct 31 04:01:35.619500 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618819    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="eabef4cb-bb40-46ef-8dfe-ce5e5df62d19" containerName="cleanup"
Oct 31 04:01:35.619500 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618824    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="eabef4cb-bb40-46ef-8dfe-ce5e5df62d19" containerName="cleanup"
Oct 31 04:01:35.619500 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618830    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4b0b6eb-4592-47d4-ba16-afa891db7458" containerName="cleanup"
Oct 31 04:01:35.619500 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618835    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44e13ed-4f1b-488c-8d16-89936835efaa" containerName="cleanup"
Oct 31 04:01:35.619500 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618841    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="044e191f-f1a6-4e6a-89e5-50da9020417d" containerName="payload-processing"
Oct 31 04:01:35.619500 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.618846    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44e13ed-4f1b-488c-8d16-89936835efaa" containerName="cleanup"
Oct 31 04:01:35.622242 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.622224    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg"
Oct 31 04:01:35.677248 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.677223    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:01:35.727395 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.727326    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 04:01:35.730527 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.730500    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg"]
Oct 31 04:01:35.736857 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.736826    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjzn5\" (UniqueName: \"kubernetes.io/projected/8c907f6c-7482-49a6-831e-2f5d3a2c1b4b-kube-api-access-wjzn5\") pod \"maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg\" (UID: \"8c907f6c-7482-49a6-831e-2f5d3a2c1b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg"
Oct 31 04:01:35.736996 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.736975    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c907f6c-7482-49a6-831e-2f5d3a2c1b4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg\" (UID: \"8c907f6c-7482-49a6-831e-2f5d3a2c1b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg"
Oct 31 04:01:35.737195 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.737173    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c907f6c-7482-49a6-831e-2f5d3a2c1b4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg\" (UID: \"8c907f6c-7482-49a6-831e-2f5d3a2c1b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg"
Oct 31 04:01:35.838761 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.838730    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c907f6c-7482-49a6-831e-2f5d3a2c1b4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg\" (UID: \"8c907f6c-7482-49a6-831e-2f5d3a2c1b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg"
Oct 31 04:01:35.839017 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.838817    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c907f6c-7482-49a6-831e-2f5d3a2c1b4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg\" (UID: \"8c907f6c-7482-49a6-831e-2f5d3a2c1b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg"
Oct 31 04:01:35.839017 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.838952    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjzn5\" (UniqueName: \"kubernetes.io/projected/8c907f6c-7482-49a6-831e-2f5d3a2c1b4b-kube-api-access-wjzn5\") pod \"maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg\" (UID: \"8c907f6c-7482-49a6-831e-2f5d3a2c1b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg"
Oct 31 04:01:35.839488 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.839461    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c907f6c-7482-49a6-831e-2f5d3a2c1b4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg\" (UID: \"8c907f6c-7482-49a6-831e-2f5d3a2c1b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg"
Oct 31 04:01:35.841293 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.841273    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c907f6c-7482-49a6-831e-2f5d3a2c1b4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg\" (UID: \"8c907f6c-7482-49a6-831e-2f5d3a2c1b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg"
Oct 31 04:01:35.900850 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.900784    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjzn5\" (UniqueName: \"kubernetes.io/projected/8c907f6c-7482-49a6-831e-2f5d3a2c1b4b-kube-api-access-wjzn5\") pod \"maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg\" (UID: \"8c907f6c-7482-49a6-831e-2f5d3a2c1b4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg"
Oct 31 04:01:35.936767 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:35.936724    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg"
Oct 31 04:01:36.316380 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:36.316355    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg"]
Oct 31 04:01:36.317814 ip-10-0-134-255 kubenswrapper[4447]: W1031 04:01:36.317789    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c907f6c_7482_49a6_831e_2f5d3a2c1b4b.slice/crio-6b0d00553c90158255d225e762974950d26d69a6ce0babd35dcccc6da8ed3916 WatchSource:0}: Error finding container 6b0d00553c90158255d225e762974950d26d69a6ce0babd35dcccc6da8ed3916: Status 404 returned error can't find the container with id 6b0d00553c90158255d225e762974950d26d69a6ce0babd35dcccc6da8ed3916
Oct 31 04:01:36.909677 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:36.909577    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg" event={"ID":"8c907f6c-7482-49a6-831e-2f5d3a2c1b4b","Type":"ContainerStarted","Data":"3db8fa67e0446a94025362e28238dc11a3821eb2e3acc74fdf91ec57276ab8eb"}
Oct 31 04:01:36.909677 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:36.909643    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg" event={"ID":"8c907f6c-7482-49a6-831e-2f5d3a2c1b4b","Type":"ContainerStarted","Data":"6b0d00553c90158255d225e762974950d26d69a6ce0babd35dcccc6da8ed3916"}
Oct 31 04:01:37.058844 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:37.058815    4447 scope.go:122] "RemoveContainer" containerID="ff24078326b5aedfab5d601e435fd5fedd23b0b2cb707a5bf3594ea21bec64e3"
Oct 31 04:01:37.060459 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:37.060415    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg" podStartSLOduration=2.060398301 podStartE2EDuration="2.060398301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:01:37.058150271 +0000 UTC m=+2522.569132650" watchObservedRunningTime="2026-10-31 04:01:37.060398301 +0000 UTC m=+2522.571380678"
Oct 31 04:01:37.917591 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:37.917558    4447 generic.go:356] "Generic (PLEG): container finished" podID="8c907f6c-7482-49a6-831e-2f5d3a2c1b4b" containerID="3db8fa67e0446a94025362e28238dc11a3821eb2e3acc74fdf91ec57276ab8eb" exitCode=0
Oct 31 04:01:37.918023 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:37.917638    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg" event={"ID":"8c907f6c-7482-49a6-831e-2f5d3a2c1b4b","Type":"ContainerDied","Data":"3db8fa67e0446a94025362e28238dc11a3821eb2e3acc74fdf91ec57276ab8eb"}
Oct 31 04:01:39.070030 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:39.070005    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg"
Oct 31 04:01:39.179378 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:39.179296    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c907f6c-7482-49a6-831e-2f5d3a2c1b4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c907f6c-7482-49a6-831e-2f5d3a2c1b4b-maas-api-xks-ca\") pod \"8c907f6c-7482-49a6-831e-2f5d3a2c1b4b\" (UID: \"8c907f6c-7482-49a6-831e-2f5d3a2c1b4b\") "
Oct 31 04:01:39.179545 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:39.179386    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c907f6c-7482-49a6-831e-2f5d3a2c1b4b-kube-api-access-wjzn5\" (UniqueName: \"kubernetes.io/projected/8c907f6c-7482-49a6-831e-2f5d3a2c1b4b-kube-api-access-wjzn5\") pod \"8c907f6c-7482-49a6-831e-2f5d3a2c1b4b\" (UID: \"8c907f6c-7482-49a6-831e-2f5d3a2c1b4b\") "
Oct 31 04:01:39.179545 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:39.179473    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c907f6c-7482-49a6-831e-2f5d3a2c1b4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c907f6c-7482-49a6-831e-2f5d3a2c1b4b-maas-api-openshift-ca\") pod \"8c907f6c-7482-49a6-831e-2f5d3a2c1b4b\" (UID: \"8c907f6c-7482-49a6-831e-2f5d3a2c1b4b\") "
Oct 31 04:01:39.179847 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:39.179811    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c907f6c-7482-49a6-831e-2f5d3a2c1b4b-maas-api-openshift-ca" pod "8c907f6c-7482-49a6-831e-2f5d3a2c1b4b" (UID: "8c907f6c-7482-49a6-831e-2f5d3a2c1b4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:01:39.181445 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:39.181424    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c907f6c-7482-49a6-831e-2f5d3a2c1b4b-maas-api-xks-ca" pod "8c907f6c-7482-49a6-831e-2f5d3a2c1b4b" (UID: "8c907f6c-7482-49a6-831e-2f5d3a2c1b4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:01:39.181620 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:39.181598    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c907f6c-7482-49a6-831e-2f5d3a2c1b4b-kube-api-access-wjzn5" pod "8c907f6c-7482-49a6-831e-2f5d3a2c1b4b" (UID: "8c907f6c-7482-49a6-831e-2f5d3a2c1b4b"). InnerVolumeSpecName "kube-api-access-wjzn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:39.281297 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:39.281262    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjzn5\" (UniqueName: \"kubernetes.io/projected/8c907f6c-7482-49a6-831e-2f5d3a2c1b4b-kube-api-access-wjzn5\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:01:39.281297 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:39.281292    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c907f6c-7482-49a6-831e-2f5d3a2c1b4b-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:01:39.281297 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:39.281301    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c907f6c-7482-49a6-831e-2f5d3a2c1b4b-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:01:39.931744 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:39.931700    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg" event={"ID":"8c907f6c-7482-49a6-831e-2f5d3a2c1b4b","Type":"ContainerDied","Data":"6b0d00553c90158255d225e762974950d26d69a6ce0babd35dcccc6da8ed3916"}
Oct 31 04:01:39.932010 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:39.931821    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b0d00553c90158255d225e762974950d26d69a6ce0babd35dcccc6da8ed3916"
Oct 31 04:01:39.932010 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:39.931960    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg"
Oct 31 04:01:40.131313 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:40.131276    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"]
Oct 31 04:01:47.213923 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:47.213872    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf"]
Oct 31 04:01:47.388226 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:47.388193    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5371165b-rrphp"]
Oct 31 04:01:47.388801 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:47.388779    4447 cpu_manager.go:414] "containerMap: removing container" podUID="8c907f6c-7482-49a6-831e-2f5d3a2c1b4b" containerName="revoke-keys"
Oct 31 04:01:47.388801 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:47.388796    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c907f6c-7482-49a6-831e-2f5d3a2c1b4b" containerName="revoke-keys"
Oct 31 04:01:47.388985 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:47.388924    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="eabef4cb-bb40-46ef-8dfe-ce5e5df62d19" containerName="cleanup"
Oct 31 04:01:47.388985 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:47.388934    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c907f6c-7482-49a6-831e-2f5d3a2c1b4b" containerName="revoke-keys"
Oct 31 04:01:47.392272 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:47.392254    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5371165b-rrphp"
Oct 31 04:01:47.439050 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:47.439017    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5371165b-rrphp"]
Oct 31 04:01:47.445959 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:47.445926    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:01:47.446169 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:47.446077    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 04:01:47.474463 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:47.474377    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bn2n\" (UniqueName: \"kubernetes.io/projected/fea93e78-e7c3-4ca6-b109-a061e9f67734-kube-api-access-2bn2n\") pod \"maas-api-revoke-sub-5371165b-rrphp\" (UID: \"fea93e78-e7c3-4ca6-b109-a061e9f67734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5371165b-rrphp"
Oct 31 04:01:47.474601 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:47.474536    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea93e78-e7c3-4ca6-b109-a061e9f67734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5371165b-rrphp\" (UID: \"fea93e78-e7c3-4ca6-b109-a061e9f67734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5371165b-rrphp"
Oct 31 04:01:47.474601 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:47.474579    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea93e78-e7c3-4ca6-b109-a061e9f67734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5371165b-rrphp\" (UID: \"fea93e78-e7c3-4ca6-b109-a061e9f67734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5371165b-rrphp"
Oct 31 04:01:47.575282 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:47.575250    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bn2n\" (UniqueName: \"kubernetes.io/projected/fea93e78-e7c3-4ca6-b109-a061e9f67734-kube-api-access-2bn2n\") pod \"maas-api-revoke-sub-5371165b-rrphp\" (UID: \"fea93e78-e7c3-4ca6-b109-a061e9f67734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5371165b-rrphp"
Oct 31 04:01:47.575481 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:47.575388    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea93e78-e7c3-4ca6-b109-a061e9f67734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5371165b-rrphp\" (UID: \"fea93e78-e7c3-4ca6-b109-a061e9f67734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5371165b-rrphp"
Oct 31 04:01:47.575481 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:47.575432    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea93e78-e7c3-4ca6-b109-a061e9f67734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5371165b-rrphp\" (UID: \"fea93e78-e7c3-4ca6-b109-a061e9f67734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5371165b-rrphp"
Oct 31 04:01:47.576224 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:47.576198    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea93e78-e7c3-4ca6-b109-a061e9f67734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5371165b-rrphp\" (UID: \"fea93e78-e7c3-4ca6-b109-a061e9f67734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5371165b-rrphp"
Oct 31 04:01:47.578126 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:47.578101    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea93e78-e7c3-4ca6-b109-a061e9f67734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5371165b-rrphp\" (UID: \"fea93e78-e7c3-4ca6-b109-a061e9f67734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5371165b-rrphp"
Oct 31 04:01:47.655009 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:47.654967    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bn2n\" (UniqueName: \"kubernetes.io/projected/fea93e78-e7c3-4ca6-b109-a061e9f67734-kube-api-access-2bn2n\") pod \"maas-api-revoke-sub-5371165b-rrphp\" (UID: \"fea93e78-e7c3-4ca6-b109-a061e9f67734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5371165b-rrphp"
Oct 31 04:01:47.705804 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:47.705769    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5371165b-rrphp"
Oct 31 04:01:48.084078 ip-10-0-134-255 kubenswrapper[4447]: W1031 04:01:48.084040    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea93e78_e7c3_4ca6_b109_a061e9f67734.slice/crio-23b54df39868a61f7394ea19d45b587d046010f404a3f5121b42e59d6a66a39e WatchSource:0}: Error finding container 23b54df39868a61f7394ea19d45b587d046010f404a3f5121b42e59d6a66a39e: Status 404 returned error can't find the container with id 23b54df39868a61f7394ea19d45b587d046010f404a3f5121b42e59d6a66a39e
Oct 31 04:01:48.084370 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:48.084346    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5371165b-rrphp"]
Oct 31 04:01:48.995486 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:48.995457    4447 generic.go:356] "Generic (PLEG): container finished" podID="fea93e78-e7c3-4ca6-b109-a061e9f67734" containerID="23f37ac3d0e265e2aab716d4aafc47fe992eb212220b12ebda52076241b3804d" exitCode=0
Oct 31 04:01:48.995919 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:48.995512    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5371165b-rrphp" event={"ID":"fea93e78-e7c3-4ca6-b109-a061e9f67734","Type":"ContainerDied","Data":"23f37ac3d0e265e2aab716d4aafc47fe992eb212220b12ebda52076241b3804d"}
Oct 31 04:01:48.995919 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:48.995552    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5371165b-rrphp" event={"ID":"fea93e78-e7c3-4ca6-b109-a061e9f67734","Type":"ContainerStarted","Data":"23b54df39868a61f7394ea19d45b587d046010f404a3f5121b42e59d6a66a39e"}
Oct 31 04:01:49.003934 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:01:49.003872    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:01:50.144802 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:50.144779    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5371165b-rrphp"
Oct 31 04:01:50.225289 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:50.225250    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea93e78-e7c3-4ca6-b109-a061e9f67734-kube-api-access-2bn2n\" (UniqueName: \"kubernetes.io/projected/fea93e78-e7c3-4ca6-b109-a061e9f67734-kube-api-access-2bn2n\") pod \"fea93e78-e7c3-4ca6-b109-a061e9f67734\" (UID: \"fea93e78-e7c3-4ca6-b109-a061e9f67734\") "
Oct 31 04:01:50.225492 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:50.225303    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fea93e78-e7c3-4ca6-b109-a061e9f67734-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea93e78-e7c3-4ca6-b109-a061e9f67734-maas-api-openshift-ca\") pod \"fea93e78-e7c3-4ca6-b109-a061e9f67734\" (UID: \"fea93e78-e7c3-4ca6-b109-a061e9f67734\") "
Oct 31 04:01:50.225492 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:50.225421    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fea93e78-e7c3-4ca6-b109-a061e9f67734-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea93e78-e7c3-4ca6-b109-a061e9f67734-maas-api-xks-ca\") pod \"fea93e78-e7c3-4ca6-b109-a061e9f67734\" (UID: \"fea93e78-e7c3-4ca6-b109-a061e9f67734\") "
Oct 31 04:01:50.225701 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:50.225677    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fea93e78-e7c3-4ca6-b109-a061e9f67734-maas-api-openshift-ca" pod "fea93e78-e7c3-4ca6-b109-a061e9f67734" (UID: "fea93e78-e7c3-4ca6-b109-a061e9f67734"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:01:50.225831 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:50.225813    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea93e78-e7c3-4ca6-b109-a061e9f67734-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:01:50.227512 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:50.227488    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fea93e78-e7c3-4ca6-b109-a061e9f67734-maas-api-xks-ca" pod "fea93e78-e7c3-4ca6-b109-a061e9f67734" (UID: "fea93e78-e7c3-4ca6-b109-a061e9f67734"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:01:50.227671 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:50.227650    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea93e78-e7c3-4ca6-b109-a061e9f67734-kube-api-access-2bn2n" pod "fea93e78-e7c3-4ca6-b109-a061e9f67734" (UID: "fea93e78-e7c3-4ca6-b109-a061e9f67734"). InnerVolumeSpecName "kube-api-access-2bn2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:50.327448 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:50.327366    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bn2n\" (UniqueName: \"kubernetes.io/projected/fea93e78-e7c3-4ca6-b109-a061e9f67734-kube-api-access-2bn2n\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:01:50.327448 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:50.327392    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea93e78-e7c3-4ca6-b109-a061e9f67734-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:01:51.008917 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:51.008873    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5371165b-rrphp"
Oct 31 04:01:51.009109 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:51.008911    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5371165b-rrphp" event={"ID":"fea93e78-e7c3-4ca6-b109-a061e9f67734","Type":"ContainerDied","Data":"23b54df39868a61f7394ea19d45b587d046010f404a3f5121b42e59d6a66a39e"}
Oct 31 04:01:51.009109 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:51.008967    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23b54df39868a61f7394ea19d45b587d046010f404a3f5121b42e59d6a66a39e"
Oct 31 04:01:52.214363 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:52.214323    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf" podUID="ade58b85-8ae0-4b92-b0f6-aef25f34878b" containerName="payload-pre-processing" containerID="cri-o://e31493a0fc40f9c9ca17c58770766d0a9d53a1fbf154bf94591444b245f04eaf" gracePeriod=25
Oct 31 04:01:55.152462 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:55.152421    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main" containerID="cri-o://faee75b798078dc1e3949ea294e2ab8726173c012afa0d22b2d7c6f132484e6e" gracePeriod=45
Oct 31 04:01:55.408722 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:55.408659    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 04:01:55.484093 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:55.484051    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a9113fb-e914-442b-aed1-e6b16edb5e91-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a9113fb-e914-442b-aed1-e6b16edb5e91-tls-certs\") pod \"6a9113fb-e914-442b-aed1-e6b16edb5e91\" (UID: \"6a9113fb-e914-442b-aed1-e6b16edb5e91\") "
Oct 31 04:01:55.484264 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:55.484123    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-home\" (UniqueName: \"kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-home\") pod \"6a9113fb-e914-442b-aed1-e6b16edb5e91\" (UID: \"6a9113fb-e914-442b-aed1-e6b16edb5e91\") "
Oct 31 04:01:55.484264 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:55.484153    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-model-cache\") pod \"6a9113fb-e914-442b-aed1-e6b16edb5e91\" (UID: \"6a9113fb-e914-442b-aed1-e6b16edb5e91\") "
Oct 31 04:01:55.484264 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:55.484246    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-tmp-dir\") pod \"6a9113fb-e914-442b-aed1-e6b16edb5e91\" (UID: \"6a9113fb-e914-442b-aed1-e6b16edb5e91\") "
Oct 31 04:01:55.484388 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:55.484297    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-dshm\") pod \"6a9113fb-e914-442b-aed1-e6b16edb5e91\" (UID: \"6a9113fb-e914-442b-aed1-e6b16edb5e91\") "
Oct 31 04:01:55.484388 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:55.484341    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a9113fb-e914-442b-aed1-e6b16edb5e91-kube-api-access-7jsq6\" (UniqueName: \"kubernetes.io/projected/6a9113fb-e914-442b-aed1-e6b16edb5e91-kube-api-access-7jsq6\") pod \"6a9113fb-e914-442b-aed1-e6b16edb5e91\" (UID: \"6a9113fb-e914-442b-aed1-e6b16edb5e91\") "
Oct 31 04:01:55.484388 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:55.484357    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-home" pod "6a9113fb-e914-442b-aed1-e6b16edb5e91" (UID: "6a9113fb-e914-442b-aed1-e6b16edb5e91"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:55.484537 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:55.484460    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-model-cache" pod "6a9113fb-e914-442b-aed1-e6b16edb5e91" (UID: "6a9113fb-e914-442b-aed1-e6b16edb5e91"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:55.484537 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:55.484500    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-tmp-dir" pod "6a9113fb-e914-442b-aed1-e6b16edb5e91" (UID: "6a9113fb-e914-442b-aed1-e6b16edb5e91"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:55.484737 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:55.484719    4447 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-home\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:01:55.484795 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:55.484744    4447 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-model-cache\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:01:55.484795 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:55.484757    4447 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-tmp-dir\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:01:55.486522 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:55.486494    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a9113fb-e914-442b-aed1-e6b16edb5e91-tls-certs" pod "6a9113fb-e914-442b-aed1-e6b16edb5e91" (UID: "6a9113fb-e914-442b-aed1-e6b16edb5e91"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:01:55.486633 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:55.486552    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-dshm" pod "6a9113fb-e914-442b-aed1-e6b16edb5e91" (UID: "6a9113fb-e914-442b-aed1-e6b16edb5e91"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:55.486728 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:55.486709    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9113fb-e914-442b-aed1-e6b16edb5e91-kube-api-access-7jsq6" pod "6a9113fb-e914-442b-aed1-e6b16edb5e91" (UID: "6a9113fb-e914-442b-aed1-e6b16edb5e91"). InnerVolumeSpecName "kube-api-access-7jsq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:55.586428 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:55.586392    4447 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a9113fb-e914-442b-aed1-e6b16edb5e91-tls-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:01:55.586428 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:55.586422    4447 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a9113fb-e914-442b-aed1-e6b16edb5e91-dshm\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:01:55.586428 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:55.586433    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jsq6\" (UniqueName: \"kubernetes.io/projected/6a9113fb-e914-442b-aed1-e6b16edb5e91-kube-api-access-7jsq6\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:01:56.040040 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:56.040004    4447 generic.go:356] "Generic (PLEG): container finished" podID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerID="faee75b798078dc1e3949ea294e2ab8726173c012afa0d22b2d7c6f132484e6e" exitCode=0
Oct 31 04:01:56.040222 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:56.040080    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"
Oct 31 04:01:56.040222 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:56.040089    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" event={"ID":"6a9113fb-e914-442b-aed1-e6b16edb5e91","Type":"ContainerDied","Data":"faee75b798078dc1e3949ea294e2ab8726173c012afa0d22b2d7c6f132484e6e"}
Oct 31 04:01:56.040222 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:56.040135    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" event={"ID":"6a9113fb-e914-442b-aed1-e6b16edb5e91","Type":"ContainerDied","Data":"8059bb561b6a83b03055284815ea744088853621647d13859bfc513bc37b9fcb"}
Oct 31 04:01:56.040222 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:56.040155    4447 scope.go:122] "RemoveContainer" containerID="faee75b798078dc1e3949ea294e2ab8726173c012afa0d22b2d7c6f132484e6e"
Oct 31 04:01:56.050442 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:56.050416    4447 scope.go:122] "RemoveContainer" containerID="faee75b798078dc1e3949ea294e2ab8726173c012afa0d22b2d7c6f132484e6e"
Oct 31 04:01:56.050706 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:01:56.050688    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faee75b798078dc1e3949ea294e2ab8726173c012afa0d22b2d7c6f132484e6e\": container with ID starting with faee75b798078dc1e3949ea294e2ab8726173c012afa0d22b2d7c6f132484e6e not found: ID does not exist" containerID="faee75b798078dc1e3949ea294e2ab8726173c012afa0d22b2d7c6f132484e6e"
Oct 31 04:01:56.050756 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:56.050714    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faee75b798078dc1e3949ea294e2ab8726173c012afa0d22b2d7c6f132484e6e"} err="failed to get container status \"faee75b798078dc1e3949ea294e2ab8726173c012afa0d22b2d7c6f132484e6e\": rpc error: code = NotFound desc = could not find container \"faee75b798078dc1e3949ea294e2ab8726173c012afa0d22b2d7c6f132484e6e\": container with ID starting with faee75b798078dc1e3949ea294e2ab8726173c012afa0d22b2d7c6f132484e6e not found: ID does not exist"
Oct 31 04:01:56.206123 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:56.206091    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"]
Oct 31 04:01:56.256369 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:56.256333    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6"]
Oct 31 04:01:56.344493 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:56.344403    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cb5a/simulator-w5-14cb5a-kserve-6d44798cdb-s9vk6" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:01:56.607959 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:01:56.607853    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" path="/var/lib/kubelet/pods/6a9113fb-e914-442b-aed1-e6b16edb5e91/volumes"
Oct 31 04:02:00.496455 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:00.496416    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf" podUID="ade58b85-8ae0-4b92-b0f6-aef25f34878b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:02:04.465053 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:04.465017    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg"]
Oct 31 04:02:04.515094 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:04.515050    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cb5a-2ttpg"]
Oct 31 04:02:04.609061 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:04.609032    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c907f6c-7482-49a6-831e-2f5d3a2c1b4b" path="/var/lib/kubelet/pods/8c907f6c-7482-49a6-831e-2f5d3a2c1b4b/volumes"
Oct 31 04:02:10.497019 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:10.496968    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf" podUID="ade58b85-8ae0-4b92-b0f6-aef25f34878b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:02:11.736964 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:11.736919    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"]
Oct 31 04:02:11.737436 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:11.737188    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch" podUID="eab02233-a047-4c35-9366-573b2dfd10cb" containerName="istio-proxy" containerID="cri-o://b8f35cb7dff26b752da6d37e184e5de01905f4b60fd566649d97363276b12584" gracePeriod=30
Oct 31 04:02:12.489519 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:12.489489    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf"
Oct 31 04:02:12.584154 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:12.584116    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade58b85-8ae0-4b92-b0f6-aef25f34878b-kube-api-access-26hp2\" (UniqueName: \"kubernetes.io/projected/ade58b85-8ae0-4b92-b0f6-aef25f34878b-kube-api-access-26hp2\") pod \"ade58b85-8ae0-4b92-b0f6-aef25f34878b\" (UID: \"ade58b85-8ae0-4b92-b0f6-aef25f34878b\") "
Oct 31 04:02:12.584347 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:12.584207    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ade58b85-8ae0-4b92-b0f6-aef25f34878b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade58b85-8ae0-4b92-b0f6-aef25f34878b-plugins-config-volume\") pod \"ade58b85-8ae0-4b92-b0f6-aef25f34878b\" (UID: \"ade58b85-8ae0-4b92-b0f6-aef25f34878b\") "
Oct 31 04:02:12.585197 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:12.585172    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade58b85-8ae0-4b92-b0f6-aef25f34878b-plugins-config-volume" pod "ade58b85-8ae0-4b92-b0f6-aef25f34878b" (UID: "ade58b85-8ae0-4b92-b0f6-aef25f34878b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:02:12.589402 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:12.589374    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade58b85-8ae0-4b92-b0f6-aef25f34878b-kube-api-access-26hp2" pod "ade58b85-8ae0-4b92-b0f6-aef25f34878b" (UID: "ade58b85-8ae0-4b92-b0f6-aef25f34878b"). InnerVolumeSpecName "kube-api-access-26hp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:02:12.687919 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:12.686491    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26hp2\" (UniqueName: \"kubernetes.io/projected/ade58b85-8ae0-4b92-b0f6-aef25f34878b-kube-api-access-26hp2\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:02:12.687919 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:12.686527    4447 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade58b85-8ae0-4b92-b0f6-aef25f34878b-plugins-config-volume\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:02:13.148757 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:13.148718    4447 generic.go:356] "Generic (PLEG): container finished" podID="ade58b85-8ae0-4b92-b0f6-aef25f34878b" containerID="e31493a0fc40f9c9ca17c58770766d0a9d53a1fbf154bf94591444b245f04eaf" exitCode=0
Oct 31 04:02:13.149207 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:13.148778    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf"
Oct 31 04:02:13.149207 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:13.148806    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf" event={"ID":"ade58b85-8ae0-4b92-b0f6-aef25f34878b","Type":"ContainerDied","Data":"e31493a0fc40f9c9ca17c58770766d0a9d53a1fbf154bf94591444b245f04eaf"}
Oct 31 04:02:13.149207 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:13.148877    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf" event={"ID":"ade58b85-8ae0-4b92-b0f6-aef25f34878b","Type":"ContainerDied","Data":"e8e65000c9649ff4942075447a24995e581e33e7a2f6664fbf41e3fc106245b6"}
Oct 31 04:02:13.149207 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:13.148914    4447 scope.go:122] "RemoveContainer" containerID="e31493a0fc40f9c9ca17c58770766d0a9d53a1fbf154bf94591444b245f04eaf"
Oct 31 04:02:13.158988 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:13.158970    4447 scope.go:122] "RemoveContainer" containerID="e31493a0fc40f9c9ca17c58770766d0a9d53a1fbf154bf94591444b245f04eaf"
Oct 31 04:02:13.159298 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:02:13.159267    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e31493a0fc40f9c9ca17c58770766d0a9d53a1fbf154bf94591444b245f04eaf\": container with ID starting with e31493a0fc40f9c9ca17c58770766d0a9d53a1fbf154bf94591444b245f04eaf not found: ID does not exist" containerID="e31493a0fc40f9c9ca17c58770766d0a9d53a1fbf154bf94591444b245f04eaf"
Oct 31 04:02:13.159348 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:13.159312    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e31493a0fc40f9c9ca17c58770766d0a9d53a1fbf154bf94591444b245f04eaf"} err="failed to get container status \"e31493a0fc40f9c9ca17c58770766d0a9d53a1fbf154bf94591444b245f04eaf\": rpc error: code = NotFound desc = could not find container \"e31493a0fc40f9c9ca17c58770766d0a9d53a1fbf154bf94591444b245f04eaf\": container with ID starting with e31493a0fc40f9c9ca17c58770766d0a9d53a1fbf154bf94591444b245f04eaf not found: ID does not exist"
Oct 31 04:02:13.314304 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:13.314270    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf"]
Oct 31 04:02:13.318738 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:13.318711    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-14cb5a-b58d4c59f-c9pmf"]
Oct 31 04:02:14.608146 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:14.608106    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ade58b85-8ae0-4b92-b0f6-aef25f34878b" path="/var/lib/kubelet/pods/ade58b85-8ae0-4b92-b0f6-aef25f34878b/volumes"
Oct 31 04:02:17.015195 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.015128    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 04:02:17.132966 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.132932    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-workload-socket\") pod \"eab02233-a047-4c35-9366-573b2dfd10cb\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") "
Oct 31 04:02:17.133151 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.133028    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-istio-envoy\") pod \"eab02233-a047-4c35-9366-573b2dfd10cb\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") "
Oct 31 04:02:17.133151 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.133053    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-workload-certs\") pod \"eab02233-a047-4c35-9366-573b2dfd10cb\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") "
Oct 31 04:02:17.133151 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.133080    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eab02233-a047-4c35-9366-573b2dfd10cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab02233-a047-4c35-9366-573b2dfd10cb-istiod-ca-cert\") pod \"eab02233-a047-4c35-9366-573b2dfd10cb\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") "
Oct 31 04:02:17.133151 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.133144    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eab02233-a047-4c35-9366-573b2dfd10cb-istio-token\" (UniqueName: \"kubernetes.io/projected/eab02233-a047-4c35-9366-573b2dfd10cb-istio-token\") pod \"eab02233-a047-4c35-9366-573b2dfd10cb\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") "
Oct 31 04:02:17.133394 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.133172    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-istio-data\") pod \"eab02233-a047-4c35-9366-573b2dfd10cb\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") "
Oct 31 04:02:17.133394 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.133206    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eab02233-a047-4c35-9366-573b2dfd10cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab02233-a047-4c35-9366-573b2dfd10cb-istio-podinfo\") pod \"eab02233-a047-4c35-9366-573b2dfd10cb\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") "
Oct 31 04:02:17.133394 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.133247    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-workload-socket" pod "eab02233-a047-4c35-9366-573b2dfd10cb" (UID: "eab02233-a047-4c35-9366-573b2dfd10cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:02:17.133394 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.133292    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-credential-socket\") pod \"eab02233-a047-4c35-9366-573b2dfd10cb\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") "
Oct 31 04:02:17.133394 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.133391    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eab02233-a047-4c35-9366-573b2dfd10cb-kube-api-access-8wzbx\" (UniqueName: \"kubernetes.io/projected/eab02233-a047-4c35-9366-573b2dfd10cb-kube-api-access-8wzbx\") pod \"eab02233-a047-4c35-9366-573b2dfd10cb\" (UID: \"eab02233-a047-4c35-9366-573b2dfd10cb\") "
Oct 31 04:02:17.133655 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.133520    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-istio-data" pod "eab02233-a047-4c35-9366-573b2dfd10cb" (UID: "eab02233-a047-4c35-9366-573b2dfd10cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:02:17.133721 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.133656    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-workload-certs" pod "eab02233-a047-4c35-9366-573b2dfd10cb" (UID: "eab02233-a047-4c35-9366-573b2dfd10cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:02:17.133975 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.133836    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-credential-socket" pod "eab02233-a047-4c35-9366-573b2dfd10cb" (UID: "eab02233-a047-4c35-9366-573b2dfd10cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:02:17.134342 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.134257    4447 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-workload-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:02:17.134342 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.134286    4447 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-istio-data\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:02:17.134342 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.134302    4447 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-credential-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:02:17.134342 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.134316    4447 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-workload-socket\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:02:17.134826 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.134797    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eab02233-a047-4c35-9366-573b2dfd10cb-istiod-ca-cert" pod "eab02233-a047-4c35-9366-573b2dfd10cb" (UID: "eab02233-a047-4c35-9366-573b2dfd10cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:02:17.135843 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.135816    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eab02233-a047-4c35-9366-573b2dfd10cb-istio-podinfo" pod "eab02233-a047-4c35-9366-573b2dfd10cb" (UID: "eab02233-a047-4c35-9366-573b2dfd10cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:02:17.135843 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.135856    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eab02233-a047-4c35-9366-573b2dfd10cb-kube-api-access-8wzbx" pod "eab02233-a047-4c35-9366-573b2dfd10cb" (UID: "eab02233-a047-4c35-9366-573b2dfd10cb"). InnerVolumeSpecName "kube-api-access-8wzbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:02:17.136105 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.135981    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-istio-envoy" pod "eab02233-a047-4c35-9366-573b2dfd10cb" (UID: "eab02233-a047-4c35-9366-573b2dfd10cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:02:17.136238 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.136211    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eab02233-a047-4c35-9366-573b2dfd10cb-istio-token" pod "eab02233-a047-4c35-9366-573b2dfd10cb" (UID: "eab02233-a047-4c35-9366-573b2dfd10cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:02:17.148533 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.148505    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-768d4c8b4f-2kskj"]
Oct 31 04:02:17.148761 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.148737    4447 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-768d4c8b4f-2kskj" podUID="2e102d26-f8d0-496a-bca4-f2d613e89745" containerName="manager" containerID="cri-o://4780a92139ba74c5277f67d2e35158a3314096b4f644e7897d7bf4ab062c527a" gracePeriod=30
Oct 31 04:02:17.177989 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.177955    4447 generic.go:356] "Generic (PLEG): container finished" podID="eab02233-a047-4c35-9366-573b2dfd10cb" containerID="b8f35cb7dff26b752da6d37e184e5de01905f4b60fd566649d97363276b12584" exitCode=0
Oct 31 04:02:17.178204 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.178002    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"
Oct 31 04:02:17.178204 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.178030    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch" event={"ID":"eab02233-a047-4c35-9366-573b2dfd10cb","Type":"ContainerDied","Data":"b8f35cb7dff26b752da6d37e184e5de01905f4b60fd566649d97363276b12584"}
Oct 31 04:02:17.178204 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.178102    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch" event={"ID":"eab02233-a047-4c35-9366-573b2dfd10cb","Type":"ContainerDied","Data":"ea6e7913a1f7f6ef2e9450dc40b3f1a3ebbe94eb9213a893d0d17ac4ec872d2a"}
Oct 31 04:02:17.178204 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.178131    4447 scope.go:122] "RemoveContainer" containerID="b8f35cb7dff26b752da6d37e184e5de01905f4b60fd566649d97363276b12584"
Oct 31 04:02:17.189173 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.189153    4447 scope.go:122] "RemoveContainer" containerID="b8f35cb7dff26b752da6d37e184e5de01905f4b60fd566649d97363276b12584"
Oct 31 04:02:17.189452 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:02:17.189429    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f35cb7dff26b752da6d37e184e5de01905f4b60fd566649d97363276b12584\": container with ID starting with b8f35cb7dff26b752da6d37e184e5de01905f4b60fd566649d97363276b12584 not found: ID does not exist" containerID="b8f35cb7dff26b752da6d37e184e5de01905f4b60fd566649d97363276b12584"
Oct 31 04:02:17.189504 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.189462    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f35cb7dff26b752da6d37e184e5de01905f4b60fd566649d97363276b12584"} err="failed to get container status \"b8f35cb7dff26b752da6d37e184e5de01905f4b60fd566649d97363276b12584\": rpc error: code = NotFound desc = could not find container \"b8f35cb7dff26b752da6d37e184e5de01905f4b60fd566649d97363276b12584\": container with ID starting with b8f35cb7dff26b752da6d37e184e5de01905f4b60fd566649d97363276b12584 not found: ID does not exist"
Oct 31 04:02:17.235492 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.235454    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wzbx\" (UniqueName: \"kubernetes.io/projected/eab02233-a047-4c35-9366-573b2dfd10cb-kube-api-access-8wzbx\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:02:17.235492 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.235491    4447 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab02233-a047-4c35-9366-573b2dfd10cb-istio-envoy\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:02:17.235685 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.235501    4447 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab02233-a047-4c35-9366-573b2dfd10cb-istiod-ca-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:02:17.235685 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.235510    4447 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eab02233-a047-4c35-9366-573b2dfd10cb-istio-token\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:02:17.235685 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.235519    4447 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab02233-a047-4c35-9366-573b2dfd10cb-istio-podinfo\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:02:17.395080 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.395054    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-768d4c8b4f-2kskj"
Oct 31 04:02:17.422388 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.422356    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"]
Oct 31 04:02:17.426565 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.426537    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cb5a-openshift-default-68676cdb8c-p6hch"]
Oct 31 04:02:17.538915 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.538854    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e102d26-f8d0-496a-bca4-f2d613e89745-kube-api-access-tvrf2\" (UniqueName: \"kubernetes.io/projected/2e102d26-f8d0-496a-bca4-f2d613e89745-kube-api-access-tvrf2\") pod \"2e102d26-f8d0-496a-bca4-f2d613e89745\" (UID: \"2e102d26-f8d0-496a-bca4-f2d613e89745\") "
Oct 31 04:02:17.539125 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.538935    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e102d26-f8d0-496a-bca4-f2d613e89745-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e102d26-f8d0-496a-bca4-f2d613e89745-metrics-certs\") pod \"2e102d26-f8d0-496a-bca4-f2d613e89745\" (UID: \"2e102d26-f8d0-496a-bca4-f2d613e89745\") "
Oct 31 04:02:17.539125 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.539039    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e102d26-f8d0-496a-bca4-f2d613e89745-webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e102d26-f8d0-496a-bca4-f2d613e89745-webhook-cert\") pod \"2e102d26-f8d0-496a-bca4-f2d613e89745\" (UID: \"2e102d26-f8d0-496a-bca4-f2d613e89745\") "
Oct 31 04:02:17.541472 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.541439    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e102d26-f8d0-496a-bca4-f2d613e89745-webhook-cert" pod "2e102d26-f8d0-496a-bca4-f2d613e89745" (UID: "2e102d26-f8d0-496a-bca4-f2d613e89745"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:02:17.542377 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.542350    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e102d26-f8d0-496a-bca4-f2d613e89745-kube-api-access-tvrf2" pod "2e102d26-f8d0-496a-bca4-f2d613e89745" (UID: "2e102d26-f8d0-496a-bca4-f2d613e89745"). InnerVolumeSpecName "kube-api-access-tvrf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:02:17.542858 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.542833    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e102d26-f8d0-496a-bca4-f2d613e89745-metrics-certs" pod "2e102d26-f8d0-496a-bca4-f2d613e89745" (UID: "2e102d26-f8d0-496a-bca4-f2d613e89745"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:02:17.640711 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.640624    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvrf2\" (UniqueName: \"kubernetes.io/projected/2e102d26-f8d0-496a-bca4-f2d613e89745-kube-api-access-tvrf2\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:02:17.640711 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.640655    4447 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e102d26-f8d0-496a-bca4-f2d613e89745-metrics-certs\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:02:17.640711 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:17.640665    4447 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e102d26-f8d0-496a-bca4-f2d613e89745-webhook-cert\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:02:18.186557 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:18.186522    4447 generic.go:356] "Generic (PLEG): container finished" podID="2e102d26-f8d0-496a-bca4-f2d613e89745" containerID="4780a92139ba74c5277f67d2e35158a3314096b4f644e7897d7bf4ab062c527a" exitCode=0
Oct 31 04:02:18.187036 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:18.186569    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-768d4c8b4f-2kskj"
Oct 31 04:02:18.187036 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:18.186610    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-768d4c8b4f-2kskj" event={"ID":"2e102d26-f8d0-496a-bca4-f2d613e89745","Type":"ContainerDied","Data":"4780a92139ba74c5277f67d2e35158a3314096b4f644e7897d7bf4ab062c527a"}
Oct 31 04:02:18.187036 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:18.186660    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-768d4c8b4f-2kskj" event={"ID":"2e102d26-f8d0-496a-bca4-f2d613e89745","Type":"ContainerDied","Data":"fe8a067f25e3a9d8a097357502e331b06f6a7003121ed6b651fe702907d2a585"}
Oct 31 04:02:18.187036 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:18.186682    4447 scope.go:122] "RemoveContainer" containerID="4780a92139ba74c5277f67d2e35158a3314096b4f644e7897d7bf4ab062c527a"
Oct 31 04:02:18.198641 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:18.198615    4447 scope.go:122] "RemoveContainer" containerID="4780a92139ba74c5277f67d2e35158a3314096b4f644e7897d7bf4ab062c527a"
Oct 31 04:02:18.198948 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:02:18.198927    4447 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4780a92139ba74c5277f67d2e35158a3314096b4f644e7897d7bf4ab062c527a\": container with ID starting with 4780a92139ba74c5277f67d2e35158a3314096b4f644e7897d7bf4ab062c527a not found: ID does not exist" containerID="4780a92139ba74c5277f67d2e35158a3314096b4f644e7897d7bf4ab062c527a"
Oct 31 04:02:18.199015 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:18.198961    4447 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4780a92139ba74c5277f67d2e35158a3314096b4f644e7897d7bf4ab062c527a"} err="failed to get container status \"4780a92139ba74c5277f67d2e35158a3314096b4f644e7897d7bf4ab062c527a\": rpc error: code = NotFound desc = could not find container \"4780a92139ba74c5277f67d2e35158a3314096b4f644e7897d7bf4ab062c527a\": container with ID starting with 4780a92139ba74c5277f67d2e35158a3314096b4f644e7897d7bf4ab062c527a not found: ID does not exist"
Oct 31 04:02:18.353298 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:18.353245    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-768d4c8b4f-2kskj"]
Oct 31 04:02:18.356805 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:18.356779    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-768d4c8b4f-2kskj"]
Oct 31 04:02:18.607180 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:18.607143    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e102d26-f8d0-496a-bca4-f2d613e89745" path="/var/lib/kubelet/pods/2e102d26-f8d0-496a-bca4-f2d613e89745/volumes"
Oct 31 04:02:18.607716 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:18.607694    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eab02233-a047-4c35-9366-573b2dfd10cb" path="/var/lib/kubelet/pods/eab02233-a047-4c35-9366-573b2dfd10cb/volumes"
Oct 31 04:02:19.002491 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:02:19.002460    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:02:37.110916 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:37.110869    4447 scope.go:122] "RemoveContainer" containerID="32e5392d04364ca73f6ac9a3da61cdc4033851d1218bd7d861077f545fbdfcd1"
Oct 31 04:02:37.122987 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:02:37.122953    4447 scope.go:122] "RemoveContainer" containerID="cb40d597a95cbd52cca69cb69c586e63ad00d306b717e8977a75e4151a8b7ae1"
Oct 31 04:02:49.003130 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:02:49.003092    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:03:19.004499 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:03:19.004464    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:03:37.181383 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:03:37.181350    4447 scope.go:122] "RemoveContainer" containerID="53f147f5034354c7d8667a89d04af3cc012fe3f83ed3af22683e86b6ee2a7ba2"
Oct 31 04:03:37.190439 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:03:37.190420    4447 scope.go:122] "RemoveContainer" containerID="86dbd2ed7737cbd0d6b4d3509e7ce9f208c04c27ddc980cb3574d6a8dc3a8603"
Oct 31 04:03:37.199064 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:03:37.199042    4447 scope.go:122] "RemoveContainer" containerID="b02d5b0e7885f13b3138723ed78fb071f6486b537d6795aadd6259af6e48b760"
Oct 31 04:03:49.065259 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:03:49.065176    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:04:05.050679 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:04:05.050642    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1794214-jcvbf"]
Oct 31 04:04:05.100411 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:04:05.100374    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1794214-jcvbf"]
Oct 31 04:04:06.607503 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:04:06.607469    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbc56ecc-85d1-4a46-b567-8ece499dba21" path="/var/lib/kubelet/pods/cbc56ecc-85d1-4a46-b567-8ece499dba21/volumes"
Oct 31 04:04:19.002237 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:04:19.002193    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:04:35.059135 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:04:35.059106    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 04:04:35.089895 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:04:35.089866    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 04:04:37.232951 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:04:37.232916    4447 scope.go:122] "RemoveContainer" containerID="d20f80c557413dfaf56c9dd8c9cc9d48b10efde114bf4bef9b289f71fb0c1151"
Oct 31 04:04:49.002420 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:04:49.002377    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:04:55.050682 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:04:55.050649    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4418c521-p8kzs"]
Oct 31 04:04:55.100315 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:04:55.100279    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4418c521-p8kzs"]
Oct 31 04:04:56.608511 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:04:56.608473    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b312195-5143-45c0-b3ac-5438944c007b" path="/var/lib/kubelet/pods/0b312195-5143-45c0-b3ac-5438944c007b/volumes"
Oct 31 04:05:19.001592 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:05:19.001510    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:05:37.260169 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:05:37.260136    4447 scope.go:122] "RemoveContainer" containerID="1539f32bbcb44b4e36c9feb563abed2c812963510f8840caaa42150a09e78fcf"
Oct 31 04:05:49.003032 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:05:49.002994    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:06:19.006143 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:06:19.006017    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:06:37.284906 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:06:37.284861    4447 scope.go:122] "RemoveContainer" containerID="d488d932039dd2ee9f5b8790a6a0192cac0fa4938f7c1746c8f91df2dad56685"
Oct 31 04:06:49.007323 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:06:49.007229    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:06:51.064413 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:06:51.064373    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5371165b-rrphp"]
Oct 31 04:06:51.130320 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:06:51.130284    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5371165b-rrphp"]
Oct 31 04:06:52.607818 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:06:52.607782    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fea93e78-e7c3-4ca6-b109-a061e9f67734" path="/var/lib/kubelet/pods/fea93e78-e7c3-4ca6-b109-a061e9f67734/volumes"
Oct 31 04:07:19.002219 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:07:19.002181    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:07:37.313209 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:07:37.313174    4447 scope.go:122] "RemoveContainer" containerID="3db8fa67e0446a94025362e28238dc11a3821eb2e3acc74fdf91ec57276ab8eb"
Oct 31 04:07:49.002330 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:07:49.002290    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:08:19.001923 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:08:19.001812    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:08:37.366709 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:08:37.366678    4447 scope.go:122] "RemoveContainer" containerID="23f37ac3d0e265e2aab716d4aafc47fe992eb212220b12ebda52076241b3804d"
Oct 31 04:08:49.061670 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:08:49.061622    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:09:19.002847 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:09:19.002810    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:09:35.097411 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:09:35.097383    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 04:09:35.134370 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:09:35.134340    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 04:09:40.751071 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:09:40.751028    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-lcwvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:09:40.751553 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:09:40.751529    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs" podUID="3d7daa66-ba76-4d1a-9de8-2a68928f22b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:09:40.881910 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:09:40.878997    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-dqwsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:09:40.881910 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:09:40.879079    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm" podUID="84553d7f-51d4-4b72-b2dd-ee522e5413ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:09:49.002696 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:09:49.002614    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:10:19.002525 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:10:19.002488    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:10:49.002484 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:10:49.002447    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:11:14.460989 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.460955    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99673acf-nt4ks"]
Oct 31 04:11:14.461509 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.461495    4447 cpu_manager.go:414] "containerMap: removing container" podUID="fea93e78-e7c3-4ca6-b109-a061e9f67734" containerName="revoke-keys"
Oct 31 04:11:14.461555 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.461510    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fea93e78-e7c3-4ca6-b109-a061e9f67734" containerName="revoke-keys"
Oct 31 04:11:14.461555 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.461524    4447 cpu_manager.go:414] "containerMap: removing container" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main"
Oct 31 04:11:14.461555 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.461529    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main"
Oct 31 04:11:14.461555 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.461543    4447 cpu_manager.go:414] "containerMap: removing container" podUID="eab02233-a047-4c35-9366-573b2dfd10cb" containerName="istio-proxy"
Oct 31 04:11:14.461555 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.461548    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eab02233-a047-4c35-9366-573b2dfd10cb" containerName="istio-proxy"
Oct 31 04:11:14.461555 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.461556    4447 cpu_manager.go:414] "containerMap: removing container" podUID="ade58b85-8ae0-4b92-b0f6-aef25f34878b" containerName="payload-pre-processing"
Oct 31 04:11:14.461742 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.461561    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade58b85-8ae0-4b92-b0f6-aef25f34878b" containerName="payload-pre-processing"
Oct 31 04:11:14.461742 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.461572    4447 cpu_manager.go:414] "containerMap: removing container" podUID="2e102d26-f8d0-496a-bca4-f2d613e89745" containerName="manager"
Oct 31 04:11:14.461742 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.461576    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e102d26-f8d0-496a-bca4-f2d613e89745" containerName="manager"
Oct 31 04:11:14.461742 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.461644    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="fea93e78-e7c3-4ca6-b109-a061e9f67734" containerName="revoke-keys"
Oct 31 04:11:14.461742 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.461651    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a9113fb-e914-442b-aed1-e6b16edb5e91" containerName="main"
Oct 31 04:11:14.461742 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.461659    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="eab02233-a047-4c35-9366-573b2dfd10cb" containerName="istio-proxy"
Oct 31 04:11:14.461742 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.461667    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade58b85-8ae0-4b92-b0f6-aef25f34878b" containerName="payload-pre-processing"
Oct 31 04:11:14.461742 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.461673    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e102d26-f8d0-496a-bca4-f2d613e89745" containerName="manager"
Oct 31 04:11:14.464950 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.464933    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99673acf-nt4ks"
Oct 31 04:11:14.517370 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.517337    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:11:14.567478 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.567443    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 04:11:14.570997 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.570973    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99673acf-nt4ks"]
Oct 31 04:11:14.625238 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.625189    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf188e2-accf-43c8-9834-fd4a9a485315-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99673acf-nt4ks\" (UID: \"dcf188e2-accf-43c8-9834-fd4a9a485315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99673acf-nt4ks"
Oct 31 04:11:14.625238 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.625246    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf188e2-accf-43c8-9834-fd4a9a485315-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99673acf-nt4ks\" (UID: \"dcf188e2-accf-43c8-9834-fd4a9a485315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99673acf-nt4ks"
Oct 31 04:11:14.625510 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.625339    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tzvg\" (UniqueName: \"kubernetes.io/projected/dcf188e2-accf-43c8-9834-fd4a9a485315-kube-api-access-6tzvg\") pod \"maas-api-revoke-sub-99673acf-nt4ks\" (UID: \"dcf188e2-accf-43c8-9834-fd4a9a485315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99673acf-nt4ks"
Oct 31 04:11:14.726072 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.725986    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf188e2-accf-43c8-9834-fd4a9a485315-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99673acf-nt4ks\" (UID: \"dcf188e2-accf-43c8-9834-fd4a9a485315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99673acf-nt4ks"
Oct 31 04:11:14.726072 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.726039    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tzvg\" (UniqueName: \"kubernetes.io/projected/dcf188e2-accf-43c8-9834-fd4a9a485315-kube-api-access-6tzvg\") pod \"maas-api-revoke-sub-99673acf-nt4ks\" (UID: \"dcf188e2-accf-43c8-9834-fd4a9a485315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99673acf-nt4ks"
Oct 31 04:11:14.726271 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.726170    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf188e2-accf-43c8-9834-fd4a9a485315-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99673acf-nt4ks\" (UID: \"dcf188e2-accf-43c8-9834-fd4a9a485315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99673acf-nt4ks"
Oct 31 04:11:14.726786 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.726763    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf188e2-accf-43c8-9834-fd4a9a485315-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99673acf-nt4ks\" (UID: \"dcf188e2-accf-43c8-9834-fd4a9a485315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99673acf-nt4ks"
Oct 31 04:11:14.728530 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.728505    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf188e2-accf-43c8-9834-fd4a9a485315-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99673acf-nt4ks\" (UID: \"dcf188e2-accf-43c8-9834-fd4a9a485315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99673acf-nt4ks"
Oct 31 04:11:14.793522 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:14.793489    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tzvg\" (UniqueName: \"kubernetes.io/projected/dcf188e2-accf-43c8-9834-fd4a9a485315-kube-api-access-6tzvg\") pod \"maas-api-revoke-sub-99673acf-nt4ks\" (UID: \"dcf188e2-accf-43c8-9834-fd4a9a485315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99673acf-nt4ks"
Oct 31 04:11:15.078482 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:15.078389    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99673acf-nt4ks"
Oct 31 04:11:15.456438 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:15.456409    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99673acf-nt4ks"]
Oct 31 04:11:15.458333 ip-10-0-134-255 kubenswrapper[4447]: W1031 04:11:15.458308    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf188e2_accf_43c8_9834_fd4a9a485315.slice/crio-ff0df1a086c47835f05b99277dd487b33601aff1dd128aac15341cb62348fd6d WatchSource:0}: Error finding container ff0df1a086c47835f05b99277dd487b33601aff1dd128aac15341cb62348fd6d: Status 404 returned error can't find the container with id ff0df1a086c47835f05b99277dd487b33601aff1dd128aac15341cb62348fd6d
Oct 31 04:11:15.605090 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:15.605043    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99673acf-nt4ks" event={"ID":"dcf188e2-accf-43c8-9834-fd4a9a485315","Type":"ContainerStarted","Data":"ff0df1a086c47835f05b99277dd487b33601aff1dd128aac15341cb62348fd6d"}
Oct 31 04:11:16.612731 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:16.612643    4447 generic.go:356] "Generic (PLEG): container finished" podID="dcf188e2-accf-43c8-9834-fd4a9a485315" containerID="428c623a90a8f78a68ed1b6c30dff1d08607b81e531fe64c2b285d1f35d68aac" exitCode=0
Oct 31 04:11:16.612731 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:16.612706    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99673acf-nt4ks" event={"ID":"dcf188e2-accf-43c8-9834-fd4a9a485315","Type":"ContainerDied","Data":"428c623a90a8f78a68ed1b6c30dff1d08607b81e531fe64c2b285d1f35d68aac"}
Oct 31 04:11:17.770733 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:17.770710    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99673acf-nt4ks"
Oct 31 04:11:17.862768 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:17.862733    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcf188e2-accf-43c8-9834-fd4a9a485315-kube-api-access-6tzvg\" (UniqueName: \"kubernetes.io/projected/dcf188e2-accf-43c8-9834-fd4a9a485315-kube-api-access-6tzvg\") pod \"dcf188e2-accf-43c8-9834-fd4a9a485315\" (UID: \"dcf188e2-accf-43c8-9834-fd4a9a485315\") "
Oct 31 04:11:17.862768 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:17.862786    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcf188e2-accf-43c8-9834-fd4a9a485315-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf188e2-accf-43c8-9834-fd4a9a485315-maas-api-openshift-ca\") pod \"dcf188e2-accf-43c8-9834-fd4a9a485315\" (UID: \"dcf188e2-accf-43c8-9834-fd4a9a485315\") "
Oct 31 04:11:17.863078 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:17.862819    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcf188e2-accf-43c8-9834-fd4a9a485315-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf188e2-accf-43c8-9834-fd4a9a485315-maas-api-xks-ca\") pod \"dcf188e2-accf-43c8-9834-fd4a9a485315\" (UID: \"dcf188e2-accf-43c8-9834-fd4a9a485315\") "
Oct 31 04:11:17.863202 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:17.863179    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf188e2-accf-43c8-9834-fd4a9a485315-maas-api-openshift-ca" pod "dcf188e2-accf-43c8-9834-fd4a9a485315" (UID: "dcf188e2-accf-43c8-9834-fd4a9a485315"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:11:17.865114 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:17.865085    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf188e2-accf-43c8-9834-fd4a9a485315-maas-api-xks-ca" pod "dcf188e2-accf-43c8-9834-fd4a9a485315" (UID: "dcf188e2-accf-43c8-9834-fd4a9a485315"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:11:17.865221 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:17.865199    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf188e2-accf-43c8-9834-fd4a9a485315-kube-api-access-6tzvg" pod "dcf188e2-accf-43c8-9834-fd4a9a485315" (UID: "dcf188e2-accf-43c8-9834-fd4a9a485315"). InnerVolumeSpecName "kube-api-access-6tzvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:11:17.964084 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:17.964046    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tzvg\" (UniqueName: \"kubernetes.io/projected/dcf188e2-accf-43c8-9834-fd4a9a485315-kube-api-access-6tzvg\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:11:17.964084 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:17.964079    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf188e2-accf-43c8-9834-fd4a9a485315-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:11:17.964313 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:17.964092    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf188e2-accf-43c8-9834-fd4a9a485315-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:11:18.627363 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:18.627338    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99673acf-nt4ks"
Oct 31 04:11:18.627505 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:18.627372    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99673acf-nt4ks" event={"ID":"dcf188e2-accf-43c8-9834-fd4a9a485315","Type":"ContainerDied","Data":"ff0df1a086c47835f05b99277dd487b33601aff1dd128aac15341cb62348fd6d"}
Oct 31 04:11:18.627505 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:18.627428    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff0df1a086c47835f05b99277dd487b33601aff1dd128aac15341cb62348fd6d"
Oct 31 04:11:19.001349 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:11:19.001317    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:11:26.698453 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:26.698422    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d"]
Oct 31 04:11:26.700936 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:26.698938    4447 cpu_manager.go:414] "containerMap: removing container" podUID="dcf188e2-accf-43c8-9834-fd4a9a485315" containerName="revoke-keys"
Oct 31 04:11:26.700936 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:26.698951    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcf188e2-accf-43c8-9834-fd4a9a485315" containerName="revoke-keys"
Oct 31 04:11:26.700936 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:26.699047    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcf188e2-accf-43c8-9834-fd4a9a485315" containerName="revoke-keys"
Oct 31 04:11:26.702168 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:26.702153    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d"
Oct 31 04:11:26.756828 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:26.756796    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c77f8f-b457-4641-aafb-7ccdb2b70715-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d\" (UID: \"79c77f8f-b457-4641-aafb-7ccdb2b70715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d"
Oct 31 04:11:26.757095 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:26.756932    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c77f8f-b457-4641-aafb-7ccdb2b70715-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d\" (UID: \"79c77f8f-b457-4641-aafb-7ccdb2b70715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d"
Oct 31 04:11:26.757095 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:26.757022    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8f8z\" (UniqueName: \"kubernetes.io/projected/79c77f8f-b457-4641-aafb-7ccdb2b70715-kube-api-access-s8f8z\") pod \"maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d\" (UID: \"79c77f8f-b457-4641-aafb-7ccdb2b70715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d"
Oct 31 04:11:26.757399 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:26.757375    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:11:26.807284 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:26.807251    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 04:11:26.810096 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:26.810074    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d"]
Oct 31 04:11:26.858662 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:26.858622    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8f8z\" (UniqueName: \"kubernetes.io/projected/79c77f8f-b457-4641-aafb-7ccdb2b70715-kube-api-access-s8f8z\") pod \"maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d\" (UID: \"79c77f8f-b457-4641-aafb-7ccdb2b70715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d"
Oct 31 04:11:26.858829 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:26.858708    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c77f8f-b457-4641-aafb-7ccdb2b70715-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d\" (UID: \"79c77f8f-b457-4641-aafb-7ccdb2b70715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d"
Oct 31 04:11:26.858829 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:26.858748    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c77f8f-b457-4641-aafb-7ccdb2b70715-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d\" (UID: \"79c77f8f-b457-4641-aafb-7ccdb2b70715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d"
Oct 31 04:11:26.859357 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:26.859337    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c77f8f-b457-4641-aafb-7ccdb2b70715-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d\" (UID: \"79c77f8f-b457-4641-aafb-7ccdb2b70715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d"
Oct 31 04:11:26.861280 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:26.861258    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c77f8f-b457-4641-aafb-7ccdb2b70715-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d\" (UID: \"79c77f8f-b457-4641-aafb-7ccdb2b70715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d"
Oct 31 04:11:26.928834 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:26.921828    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8f8z\" (UniqueName: \"kubernetes.io/projected/79c77f8f-b457-4641-aafb-7ccdb2b70715-kube-api-access-s8f8z\") pod \"maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d\" (UID: \"79c77f8f-b457-4641-aafb-7ccdb2b70715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d"
Oct 31 04:11:27.016085 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:27.016045    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d"
Oct 31 04:11:27.391058 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:27.391029    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d"]
Oct 31 04:11:27.392418 ip-10-0-134-255 kubenswrapper[4447]: W1031 04:11:27.392375    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c77f8f_b457_4641_aafb_7ccdb2b70715.slice/crio-e669b46ccbfb679f95336941056333a5f6b5c1d15c25e2e038f77a43fd7beed3 WatchSource:0}: Error finding container e669b46ccbfb679f95336941056333a5f6b5c1d15c25e2e038f77a43fd7beed3: Status 404 returned error can't find the container with id e669b46ccbfb679f95336941056333a5f6b5c1d15c25e2e038f77a43fd7beed3
Oct 31 04:11:27.680704 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:27.680664    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d" event={"ID":"79c77f8f-b457-4641-aafb-7ccdb2b70715","Type":"ContainerStarted","Data":"e669b46ccbfb679f95336941056333a5f6b5c1d15c25e2e038f77a43fd7beed3"}
Oct 31 04:11:28.689194 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:28.689107    4447 generic.go:356] "Generic (PLEG): container finished" podID="79c77f8f-b457-4641-aafb-7ccdb2b70715" containerID="7013a88a95e529c6c4121e608800fd45ca7268d6ae1ebf2a8782eb09f241394d" exitCode=0
Oct 31 04:11:28.689568 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:28.689197    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d" event={"ID":"79c77f8f-b457-4641-aafb-7ccdb2b70715","Type":"ContainerDied","Data":"7013a88a95e529c6c4121e608800fd45ca7268d6ae1ebf2a8782eb09f241394d"}
Oct 31 04:11:29.858373 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:29.858350    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d"
Oct 31 04:11:29.990842 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:29.990808    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79c77f8f-b457-4641-aafb-7ccdb2b70715-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c77f8f-b457-4641-aafb-7ccdb2b70715-maas-api-xks-ca\") pod \"79c77f8f-b457-4641-aafb-7ccdb2b70715\" (UID: \"79c77f8f-b457-4641-aafb-7ccdb2b70715\") "
Oct 31 04:11:29.991052 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:29.990856    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79c77f8f-b457-4641-aafb-7ccdb2b70715-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c77f8f-b457-4641-aafb-7ccdb2b70715-maas-api-openshift-ca\") pod \"79c77f8f-b457-4641-aafb-7ccdb2b70715\" (UID: \"79c77f8f-b457-4641-aafb-7ccdb2b70715\") "
Oct 31 04:11:29.991052 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:29.991011    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c77f8f-b457-4641-aafb-7ccdb2b70715-kube-api-access-s8f8z\" (UniqueName: \"kubernetes.io/projected/79c77f8f-b457-4641-aafb-7ccdb2b70715-kube-api-access-s8f8z\") pod \"79c77f8f-b457-4641-aafb-7ccdb2b70715\" (UID: \"79c77f8f-b457-4641-aafb-7ccdb2b70715\") "
Oct 31 04:11:29.991297 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:29.991239    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c77f8f-b457-4641-aafb-7ccdb2b70715-maas-api-openshift-ca" pod "79c77f8f-b457-4641-aafb-7ccdb2b70715" (UID: "79c77f8f-b457-4641-aafb-7ccdb2b70715"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:11:29.991524 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:29.991506    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79c77f8f-b457-4641-aafb-7ccdb2b70715-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:11:29.993330 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:29.993300    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c77f8f-b457-4641-aafb-7ccdb2b70715-kube-api-access-s8f8z" pod "79c77f8f-b457-4641-aafb-7ccdb2b70715" (UID: "79c77f8f-b457-4641-aafb-7ccdb2b70715"). InnerVolumeSpecName "kube-api-access-s8f8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:11:29.993449 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:29.993400    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c77f8f-b457-4641-aafb-7ccdb2b70715-maas-api-xks-ca" pod "79c77f8f-b457-4641-aafb-7ccdb2b70715" (UID: "79c77f8f-b457-4641-aafb-7ccdb2b70715"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:11:30.092709 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:30.092676    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79c77f8f-b457-4641-aafb-7ccdb2b70715-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:11:30.092709 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:30.092703    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8f8z\" (UniqueName: \"kubernetes.io/projected/79c77f8f-b457-4641-aafb-7ccdb2b70715-kube-api-access-s8f8z\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:11:30.701679 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:30.701644    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d"
Oct 31 04:11:30.701679 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:30.701670    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d" event={"ID":"79c77f8f-b457-4641-aafb-7ccdb2b70715","Type":"ContainerDied","Data":"e669b46ccbfb679f95336941056333a5f6b5c1d15c25e2e038f77a43fd7beed3"}
Oct 31 04:11:30.701925 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:30.701720    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e669b46ccbfb679f95336941056333a5f6b5c1d15c25e2e038f77a43fd7beed3"
Oct 31 04:11:47.036958 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:47.036913    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d"]
Oct 31 04:11:47.084517 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:47.084477    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-20cd9e-llx9d"]
Oct 31 04:11:48.607425 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:11:48.607391    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79c77f8f-b457-4641-aafb-7ccdb2b70715" path="/var/lib/kubelet/pods/79c77f8f-b457-4641-aafb-7ccdb2b70715/volumes"
Oct 31 04:11:49.002839 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:11:49.002805    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:12:19.002565 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:12:19.002526    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:12:49.002465 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:12:49.002381    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:13:19.001538 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:13:19.001503    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:13:49.001803 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:13:49.001768    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:14:19.002342 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:14:19.002253    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:14:35.139704 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:14:35.139676    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 04:14:35.176153 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:14:35.176127    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 04:14:49.002326 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:14:49.002292    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:15:00.157680 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:00.157644    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr"]
Oct 31 04:15:00.158203 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:00.158180    4447 cpu_manager.go:414] "containerMap: removing container" podUID="79c77f8f-b457-4641-aafb-7ccdb2b70715" containerName="revoke-keys"
Oct 31 04:15:00.158203 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:00.158193    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c77f8f-b457-4641-aafb-7ccdb2b70715" containerName="revoke-keys"
Oct 31 04:15:00.158295 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:00.158285    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="79c77f8f-b457-4641-aafb-7ccdb2b70715" containerName="revoke-keys"
Oct 31 04:15:00.161499 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:00.161482    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr"
Oct 31 04:15:00.260783 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:00.260746    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq6wh\" (UniqueName: \"kubernetes.io/projected/1da3b3b6-ddd9-408e-a483-55a81ab12efc-kube-api-access-zq6wh\") pod \"maas-api-key-cleanup-29890335-ccmmr\" (UID: \"1da3b3b6-ddd9-408e-a483-55a81ab12efc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr"
Oct 31 04:15:00.260993 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:00.260800    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da3b3b6-ddd9-408e-a483-55a81ab12efc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890335-ccmmr\" (UID: \"1da3b3b6-ddd9-408e-a483-55a81ab12efc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr"
Oct 31 04:15:00.265520 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:00.265493    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 04:15:00.268640 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:00.268611    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr"]
Oct 31 04:15:00.361612 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:00.361575    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq6wh\" (UniqueName: \"kubernetes.io/projected/1da3b3b6-ddd9-408e-a483-55a81ab12efc-kube-api-access-zq6wh\") pod \"maas-api-key-cleanup-29890335-ccmmr\" (UID: \"1da3b3b6-ddd9-408e-a483-55a81ab12efc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr"
Oct 31 04:15:00.361796 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:00.361619    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da3b3b6-ddd9-408e-a483-55a81ab12efc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890335-ccmmr\" (UID: \"1da3b3b6-ddd9-408e-a483-55a81ab12efc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr"
Oct 31 04:15:00.362316 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:00.362295    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da3b3b6-ddd9-408e-a483-55a81ab12efc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890335-ccmmr\" (UID: \"1da3b3b6-ddd9-408e-a483-55a81ab12efc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr"
Oct 31 04:15:00.422964 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:00.422869    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq6wh\" (UniqueName: \"kubernetes.io/projected/1da3b3b6-ddd9-408e-a483-55a81ab12efc-kube-api-access-zq6wh\") pod \"maas-api-key-cleanup-29890335-ccmmr\" (UID: \"1da3b3b6-ddd9-408e-a483-55a81ab12efc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr"
Oct 31 04:15:00.474652 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:00.474616    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr"
Oct 31 04:15:00.849413 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:00.849381    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr"]
Oct 31 04:15:00.851192 ip-10-0-134-255 kubenswrapper[4447]: W1031 04:15:00.851161    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da3b3b6_ddd9_408e_a483_55a81ab12efc.slice/crio-b346234629119ffd68d33c97638a4ff3fb53316537dcd54df43cd65fdf2e7eb0 WatchSource:0}: Error finding container b346234629119ffd68d33c97638a4ff3fb53316537dcd54df43cd65fdf2e7eb0: Status 404 returned error can't find the container with id b346234629119ffd68d33c97638a4ff3fb53316537dcd54df43cd65fdf2e7eb0
Oct 31 04:15:00.985779 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:00.985746    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr" event={"ID":"1da3b3b6-ddd9-408e-a483-55a81ab12efc","Type":"ContainerStarted","Data":"b346234629119ffd68d33c97638a4ff3fb53316537dcd54df43cd65fdf2e7eb0"}
Oct 31 04:15:01.993156 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:01.993124    4447 generic.go:356] "Generic (PLEG): container finished" podID="1da3b3b6-ddd9-408e-a483-55a81ab12efc" containerID="aa66bc123aa8ea8a9bb9504c6e2915b39c158df155d8a4953018a5b65aac3e72" exitCode=60
Oct 31 04:15:01.993565 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:01.993226    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr" event={"ID":"1da3b3b6-ddd9-408e-a483-55a81ab12efc","Type":"ContainerDied","Data":"aa66bc123aa8ea8a9bb9504c6e2915b39c158df155d8a4953018a5b65aac3e72"}
Oct 31 04:15:01.993565 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:01.993455    4447 scope.go:122] "RemoveContainer" containerID="aa66bc123aa8ea8a9bb9504c6e2915b39c158df155d8a4953018a5b65aac3e72"
Oct 31 04:15:03.001854 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:03.001819    4447 generic.go:356] "Generic (PLEG): container finished" podID="1da3b3b6-ddd9-408e-a483-55a81ab12efc" containerID="ae6a5e588e726103b8baeb6e314bf8a94d6a88a90ef721dd6ef42fc9ba85e43b" exitCode=60
Oct 31 04:15:03.002379 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:03.001878    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr" event={"ID":"1da3b3b6-ddd9-408e-a483-55a81ab12efc","Type":"ContainerDied","Data":"ae6a5e588e726103b8baeb6e314bf8a94d6a88a90ef721dd6ef42fc9ba85e43b"}
Oct 31 04:15:03.002379 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:03.001997    4447 scope.go:122] "RemoveContainer" containerID="aa66bc123aa8ea8a9bb9504c6e2915b39c158df155d8a4953018a5b65aac3e72"
Oct 31 04:15:03.002379 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:03.002283    4447 scope.go:122] "RemoveContainer" containerID="ae6a5e588e726103b8baeb6e314bf8a94d6a88a90ef721dd6ef42fc9ba85e43b"
Oct 31 04:15:03.002565 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:15:03.002543    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890335-ccmmr_odh-ai-gateway-infra(1da3b3b6-ddd9-408e-a483-55a81ab12efc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr" podUID="1da3b3b6-ddd9-408e-a483-55a81ab12efc"
Oct 31 04:15:04.009499 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:04.009461    4447 scope.go:122] "RemoveContainer" containerID="ae6a5e588e726103b8baeb6e314bf8a94d6a88a90ef721dd6ef42fc9ba85e43b"
Oct 31 04:15:04.009918 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:15:04.009712    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890335-ccmmr_odh-ai-gateway-infra(1da3b3b6-ddd9-408e-a483-55a81ab12efc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr" podUID="1da3b3b6-ddd9-408e-a483-55a81ab12efc"
Oct 31 04:15:17.600650 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:17.600613    4447 scope.go:122] "RemoveContainer" containerID="ae6a5e588e726103b8baeb6e314bf8a94d6a88a90ef721dd6ef42fc9ba85e43b"
Oct 31 04:15:18.116659 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:18.116623    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr" event={"ID":"1da3b3b6-ddd9-408e-a483-55a81ab12efc","Type":"ContainerStarted","Data":"e4496a79ca9955455617d4692e83eaf491f42d766eeb45f3326133730bb5c7fb"}
Oct 31 04:15:19.001459 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:15:19.001418    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:15:19.124534 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:19.124502    4447 generic.go:356] "Generic (PLEG): container finished" podID="1da3b3b6-ddd9-408e-a483-55a81ab12efc" containerID="e4496a79ca9955455617d4692e83eaf491f42d766eeb45f3326133730bb5c7fb" exitCode=60
Oct 31 04:15:19.124726 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:19.124576    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr" event={"ID":"1da3b3b6-ddd9-408e-a483-55a81ab12efc","Type":"ContainerDied","Data":"e4496a79ca9955455617d4692e83eaf491f42d766eeb45f3326133730bb5c7fb"}
Oct 31 04:15:19.124726 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:19.124634    4447 scope.go:122] "RemoveContainer" containerID="ae6a5e588e726103b8baeb6e314bf8a94d6a88a90ef721dd6ef42fc9ba85e43b"
Oct 31 04:15:19.125068 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:19.125049    4447 scope.go:122] "RemoveContainer" containerID="e4496a79ca9955455617d4692e83eaf491f42d766eeb45f3326133730bb5c7fb"
Oct 31 04:15:19.125331 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:15:19.125313    4447 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890335-ccmmr_odh-ai-gateway-infra(1da3b3b6-ddd9-408e-a483-55a81ab12efc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr" podUID="1da3b3b6-ddd9-408e-a483-55a81ab12efc"
Oct 31 04:15:19.255806 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:19.255698    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr" podStartSLOduration=19.255682643 podStartE2EDuration="19.255682643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:18.234308012 +0000 UTC m=+3343.745290391" watchObservedRunningTime="2026-10-31 04:15:19.255682643 +0000 UTC m=+3344.766665060"
Oct 31 04:15:19.256811 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:19.256791    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr"]
Oct 31 04:15:20.275396 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:20.275366    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr"
Oct 31 04:15:20.390507 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:20.390423    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1da3b3b6-ddd9-408e-a483-55a81ab12efc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da3b3b6-ddd9-408e-a483-55a81ab12efc-maas-api-service-ca\") pod \"1da3b3b6-ddd9-408e-a483-55a81ab12efc\" (UID: \"1da3b3b6-ddd9-408e-a483-55a81ab12efc\") "
Oct 31 04:15:20.390507 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:20.390478    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1da3b3b6-ddd9-408e-a483-55a81ab12efc-kube-api-access-zq6wh\" (UniqueName: \"kubernetes.io/projected/1da3b3b6-ddd9-408e-a483-55a81ab12efc-kube-api-access-zq6wh\") pod \"1da3b3b6-ddd9-408e-a483-55a81ab12efc\" (UID: \"1da3b3b6-ddd9-408e-a483-55a81ab12efc\") "
Oct 31 04:15:20.390801 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:20.390776    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da3b3b6-ddd9-408e-a483-55a81ab12efc-maas-api-service-ca" pod "1da3b3b6-ddd9-408e-a483-55a81ab12efc" (UID: "1da3b3b6-ddd9-408e-a483-55a81ab12efc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:20.392674 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:20.392646    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da3b3b6-ddd9-408e-a483-55a81ab12efc-kube-api-access-zq6wh" pod "1da3b3b6-ddd9-408e-a483-55a81ab12efc" (UID: "1da3b3b6-ddd9-408e-a483-55a81ab12efc"). InnerVolumeSpecName "kube-api-access-zq6wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:20.491690 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:20.491654    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da3b3b6-ddd9-408e-a483-55a81ab12efc-maas-api-service-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:15:20.491690 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:20.491684    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq6wh\" (UniqueName: \"kubernetes.io/projected/1da3b3b6-ddd9-408e-a483-55a81ab12efc-kube-api-access-zq6wh\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:15:21.139164 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:21.139128    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr"
Oct 31 04:15:21.139374 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:21.139166    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr" event={"ID":"1da3b3b6-ddd9-408e-a483-55a81ab12efc","Type":"ContainerDied","Data":"b346234629119ffd68d33c97638a4ff3fb53316537dcd54df43cd65fdf2e7eb0"}
Oct 31 04:15:21.139374 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:21.139236    4447 scope.go:122] "RemoveContainer" containerID="e4496a79ca9955455617d4692e83eaf491f42d766eeb45f3326133730bb5c7fb"
Oct 31 04:15:21.304483 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:21.304423    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr"]
Oct 31 04:15:21.354697 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:21.354654    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890335-ccmmr"]
Oct 31 04:15:22.607827 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:15:22.607796    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1da3b3b6-ddd9-408e-a483-55a81ab12efc" path="/var/lib/kubelet/pods/1da3b3b6-ddd9-408e-a483-55a81ab12efc/volumes"
Oct 31 04:15:49.002544 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:15:49.002465    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:16:04.525062 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:04.525024    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc4ca437-9shcm"]
Oct 31 04:16:04.525537 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:04.525524    4447 cpu_manager.go:414] "containerMap: removing container" podUID="1da3b3b6-ddd9-408e-a483-55a81ab12efc" containerName="cleanup"
Oct 31 04:16:04.525593 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:04.525538    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1da3b3b6-ddd9-408e-a483-55a81ab12efc" containerName="cleanup"
Oct 31 04:16:04.525593 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:04.525553    4447 cpu_manager.go:414] "containerMap: removing container" podUID="1da3b3b6-ddd9-408e-a483-55a81ab12efc" containerName="cleanup"
Oct 31 04:16:04.525593 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:04.525559    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1da3b3b6-ddd9-408e-a483-55a81ab12efc" containerName="cleanup"
Oct 31 04:16:04.525704 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:04.525644    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="1da3b3b6-ddd9-408e-a483-55a81ab12efc" containerName="cleanup"
Oct 31 04:16:04.525704 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:04.525651    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="1da3b3b6-ddd9-408e-a483-55a81ab12efc" containerName="cleanup"
Oct 31 04:16:04.530026 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:04.530008    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4ca437-9shcm"
Oct 31 04:16:04.586916 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:04.586870    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc4ca437-9shcm"]
Oct 31 04:16:04.596846 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:04.596804    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc4ca437-9shcm\" (UID: \"c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4ca437-9shcm"
Oct 31 04:16:04.597229 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:04.597200    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdv94\" (UniqueName: \"kubernetes.io/projected/c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a-kube-api-access-sdv94\") pod \"maas-api-revoke-sub-cc4ca437-9shcm\" (UID: \"c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4ca437-9shcm"
Oct 31 04:16:04.597361 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:04.597279    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc4ca437-9shcm\" (UID: \"c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4ca437-9shcm"
Oct 31 04:16:04.632759 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:04.632723    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:16:04.632759 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:04.632735    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 04:16:04.698166 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:04.698130    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc4ca437-9shcm\" (UID: \"c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4ca437-9shcm"
Oct 31 04:16:04.698386 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:04.698201    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdv94\" (UniqueName: \"kubernetes.io/projected/c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a-kube-api-access-sdv94\") pod \"maas-api-revoke-sub-cc4ca437-9shcm\" (UID: \"c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4ca437-9shcm"
Oct 31 04:16:04.698386 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:04.698265    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc4ca437-9shcm\" (UID: \"c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4ca437-9shcm"
Oct 31 04:16:04.698852 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:04.698820    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc4ca437-9shcm\" (UID: \"c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4ca437-9shcm"
Oct 31 04:16:04.700985 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:04.700959    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc4ca437-9shcm\" (UID: \"c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4ca437-9shcm"
Oct 31 04:16:04.759119 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:04.759091    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdv94\" (UniqueName: \"kubernetes.io/projected/c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a-kube-api-access-sdv94\") pod \"maas-api-revoke-sub-cc4ca437-9shcm\" (UID: \"c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4ca437-9shcm"
Oct 31 04:16:04.842726 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:04.842646    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4ca437-9shcm"
Oct 31 04:16:05.217279 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:05.217248    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc4ca437-9shcm"]
Oct 31 04:16:05.218761 ip-10-0-134-255 kubenswrapper[4447]: W1031 04:16:05.218734    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d40eb9_55c3_41e9_8c89_ba1dbe509c6a.slice/crio-ae5361ca5a8563f4b611254f00be42978aa874e6e70881141a612a0233cbd9d2 WatchSource:0}: Error finding container ae5361ca5a8563f4b611254f00be42978aa874e6e70881141a612a0233cbd9d2: Status 404 returned error can't find the container with id ae5361ca5a8563f4b611254f00be42978aa874e6e70881141a612a0233cbd9d2
Oct 31 04:16:05.391685 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:05.391651    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4ca437-9shcm" event={"ID":"c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a","Type":"ContainerStarted","Data":"ae5361ca5a8563f4b611254f00be42978aa874e6e70881141a612a0233cbd9d2"}
Oct 31 04:16:06.403794 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:06.403695    4447 generic.go:356] "Generic (PLEG): container finished" podID="c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a" containerID="9fefcc9f0150ad5d0941600f915a828dfe01756bd0d55787cb8ae926a7ffec81" exitCode=0
Oct 31 04:16:06.403794 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:06.403746    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4ca437-9shcm" event={"ID":"c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a","Type":"ContainerDied","Data":"9fefcc9f0150ad5d0941600f915a828dfe01756bd0d55787cb8ae926a7ffec81"}
Oct 31 04:16:07.550143 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:07.550119    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4ca437-9shcm"
Oct 31 04:16:07.629684 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:07.629649    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a-maas-api-openshift-ca\") pod \"c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a\" (UID: \"c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a\") "
Oct 31 04:16:07.629879 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:07.629710    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a-maas-api-xks-ca\") pod \"c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a\" (UID: \"c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a\") "
Oct 31 04:16:07.629879 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:07.629774    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a-kube-api-access-sdv94\" (UniqueName: \"kubernetes.io/projected/c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a-kube-api-access-sdv94\") pod \"c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a\" (UID: \"c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a\") "
Oct 31 04:16:07.630077 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:07.630045    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a-maas-api-openshift-ca" pod "c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a" (UID: "c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:16:07.630204 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:07.630137    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:16:07.632113 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:07.632089    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a-kube-api-access-sdv94" pod "c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a" (UID: "c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a"). InnerVolumeSpecName "kube-api-access-sdv94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:16:07.632244 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:07.632224    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a-maas-api-xks-ca" pod "c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a" (UID: "c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:16:07.731143 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:07.731102    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdv94\" (UniqueName: \"kubernetes.io/projected/c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a-kube-api-access-sdv94\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:16:07.731143 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:07.731134    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:16:08.417612 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:08.417575    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4ca437-9shcm" event={"ID":"c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a","Type":"ContainerDied","Data":"ae5361ca5a8563f4b611254f00be42978aa874e6e70881141a612a0233cbd9d2"}
Oct 31 04:16:08.417612 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:08.417600    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc4ca437-9shcm"
Oct 31 04:16:08.417936 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:08.417632    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae5361ca5a8563f4b611254f00be42978aa874e6e70881141a612a0233cbd9d2"
Oct 31 04:16:16.097250 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:16.097215    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48"]
Oct 31 04:16:16.097713 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:16.097700    4447 cpu_manager.go:414] "containerMap: removing container" podUID="c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a" containerName="revoke-keys"
Oct 31 04:16:16.097760 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:16.097714    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a" containerName="revoke-keys"
Oct 31 04:16:16.097760 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:16.097732    4447 cpu_manager.go:414] "containerMap: removing container" podUID="1da3b3b6-ddd9-408e-a483-55a81ab12efc" containerName="cleanup"
Oct 31 04:16:16.097760 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:16.097737    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1da3b3b6-ddd9-408e-a483-55a81ab12efc" containerName="cleanup"
Oct 31 04:16:16.097857 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:16.097830    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a" containerName="revoke-keys"
Oct 31 04:16:16.097857 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:16.097837    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="1da3b3b6-ddd9-408e-a483-55a81ab12efc" containerName="cleanup"
Oct 31 04:16:16.101228 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:16.101207    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48"
Oct 31 04:16:16.160555 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:16.160521    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48"]
Oct 31 04:16:16.205927 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:16.205892    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:16:16.205927 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:16.205921    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zpr9\""
Oct 31 04:16:16.223114 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:16.223077    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f986ba-aa88-4896-9e16-c1701225cb47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48\" (UID: \"95f986ba-aa88-4896-9e16-c1701225cb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48"
Oct 31 04:16:16.223292 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:16.223139    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7ggx\" (UniqueName: \"kubernetes.io/projected/95f986ba-aa88-4896-9e16-c1701225cb47-kube-api-access-g7ggx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48\" (UID: \"95f986ba-aa88-4896-9e16-c1701225cb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48"
Oct 31 04:16:16.223292 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:16.223241    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f986ba-aa88-4896-9e16-c1701225cb47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48\" (UID: \"95f986ba-aa88-4896-9e16-c1701225cb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48"
Oct 31 04:16:16.324353 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:16.324312    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f986ba-aa88-4896-9e16-c1701225cb47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48\" (UID: \"95f986ba-aa88-4896-9e16-c1701225cb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48"
Oct 31 04:16:16.324567 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:16.324375    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7ggx\" (UniqueName: \"kubernetes.io/projected/95f986ba-aa88-4896-9e16-c1701225cb47-kube-api-access-g7ggx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48\" (UID: \"95f986ba-aa88-4896-9e16-c1701225cb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48"
Oct 31 04:16:16.324567 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:16.324421    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f986ba-aa88-4896-9e16-c1701225cb47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48\" (UID: \"95f986ba-aa88-4896-9e16-c1701225cb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48"
Oct 31 04:16:16.325021 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:16.324998    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f986ba-aa88-4896-9e16-c1701225cb47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48\" (UID: \"95f986ba-aa88-4896-9e16-c1701225cb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48"
Oct 31 04:16:16.327008 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:16.326984    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f986ba-aa88-4896-9e16-c1701225cb47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48\" (UID: \"95f986ba-aa88-4896-9e16-c1701225cb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48"
Oct 31 04:16:16.385523 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:16.385441    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7ggx\" (UniqueName: \"kubernetes.io/projected/95f986ba-aa88-4896-9e16-c1701225cb47-kube-api-access-g7ggx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48\" (UID: \"95f986ba-aa88-4896-9e16-c1701225cb47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48"
Oct 31 04:16:16.414381 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:16.414347    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48"
Oct 31 04:16:16.791111 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:16.791084    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48"]
Oct 31 04:16:16.793321 ip-10-0-134-255 kubenswrapper[4447]: W1031 04:16:16.793295    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f986ba_aa88_4896_9e16_c1701225cb47.slice/crio-9ee5de18997512b3130b005bf3d1cd9d23e635568eaff535f4da938fb86f1a24 WatchSource:0}: Error finding container 9ee5de18997512b3130b005bf3d1cd9d23e635568eaff535f4da938fb86f1a24: Status 404 returned error can't find the container with id 9ee5de18997512b3130b005bf3d1cd9d23e635568eaff535f4da938fb86f1a24
Oct 31 04:16:17.489396 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:17.489358    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48" event={"ID":"95f986ba-aa88-4896-9e16-c1701225cb47","Type":"ContainerStarted","Data":"536c8b472579513f79bcee66e57e3f9963b470e0b12fe4a71cfa40c6de2b4aa0"}
Oct 31 04:16:17.489805 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:17.489414    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48" event={"ID":"95f986ba-aa88-4896-9e16-c1701225cb47","Type":"ContainerStarted","Data":"9ee5de18997512b3130b005bf3d1cd9d23e635568eaff535f4da938fb86f1a24"}
Oct 31 04:16:17.607475 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:17.607427    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48" podStartSLOduration=1.607415102 podStartE2EDuration="1.607415102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:16:17.605119618 +0000 UTC m=+3403.116101997" watchObservedRunningTime="2026-10-31 04:16:17.607415102 +0000 UTC m=+3403.118397513"
Oct 31 04:16:18.047702 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:18.047665    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99673acf-nt4ks"]
Oct 31 04:16:18.050237 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:18.050214    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99673acf-nt4ks"]
Oct 31 04:16:18.497409 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:18.497375    4447 generic.go:356] "Generic (PLEG): container finished" podID="95f986ba-aa88-4896-9e16-c1701225cb47" containerID="536c8b472579513f79bcee66e57e3f9963b470e0b12fe4a71cfa40c6de2b4aa0" exitCode=0
Oct 31 04:16:18.497819 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:18.497461    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48" event={"ID":"95f986ba-aa88-4896-9e16-c1701225cb47","Type":"ContainerDied","Data":"536c8b472579513f79bcee66e57e3f9963b470e0b12fe4a71cfa40c6de2b4aa0"}
Oct 31 04:16:18.607914 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:18.607844    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcf188e2-accf-43c8-9834-fd4a9a485315" path="/var/lib/kubelet/pods/dcf188e2-accf-43c8-9834-fd4a9a485315/volumes"
Oct 31 04:16:19.002224 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:16:19.002184    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:16:19.648861 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:19.648837    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48"
Oct 31 04:16:19.775806 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:19.775716    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95f986ba-aa88-4896-9e16-c1701225cb47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f986ba-aa88-4896-9e16-c1701225cb47-maas-api-xks-ca\") pod \"95f986ba-aa88-4896-9e16-c1701225cb47\" (UID: \"95f986ba-aa88-4896-9e16-c1701225cb47\") "
Oct 31 04:16:19.775992 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:19.775863    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95f986ba-aa88-4896-9e16-c1701225cb47-kube-api-access-g7ggx\" (UniqueName: \"kubernetes.io/projected/95f986ba-aa88-4896-9e16-c1701225cb47-kube-api-access-g7ggx\") pod \"95f986ba-aa88-4896-9e16-c1701225cb47\" (UID: \"95f986ba-aa88-4896-9e16-c1701225cb47\") "
Oct 31 04:16:19.775992 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:19.775932    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95f986ba-aa88-4896-9e16-c1701225cb47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f986ba-aa88-4896-9e16-c1701225cb47-maas-api-openshift-ca\") pod \"95f986ba-aa88-4896-9e16-c1701225cb47\" (UID: \"95f986ba-aa88-4896-9e16-c1701225cb47\") "
Oct 31 04:16:19.776324 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:19.776295    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95f986ba-aa88-4896-9e16-c1701225cb47-maas-api-openshift-ca" pod "95f986ba-aa88-4896-9e16-c1701225cb47" (UID: "95f986ba-aa88-4896-9e16-c1701225cb47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:16:19.777154 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:19.776996    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f986ba-aa88-4896-9e16-c1701225cb47-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:16:19.778265 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:19.778237    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95f986ba-aa88-4896-9e16-c1701225cb47-maas-api-xks-ca" pod "95f986ba-aa88-4896-9e16-c1701225cb47" (UID: "95f986ba-aa88-4896-9e16-c1701225cb47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:16:19.778391 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:19.778315    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f986ba-aa88-4896-9e16-c1701225cb47-kube-api-access-g7ggx" pod "95f986ba-aa88-4896-9e16-c1701225cb47" (UID: "95f986ba-aa88-4896-9e16-c1701225cb47"). InnerVolumeSpecName "kube-api-access-g7ggx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:16:19.878564 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:19.878527    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7ggx\" (UniqueName: \"kubernetes.io/projected/95f986ba-aa88-4896-9e16-c1701225cb47-kube-api-access-g7ggx\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:16:19.878564 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:19.878559    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f986ba-aa88-4896-9e16-c1701225cb47-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:16:20.511153 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:20.511124    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48"
Oct 31 04:16:20.511324 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:20.511150    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48" event={"ID":"95f986ba-aa88-4896-9e16-c1701225cb47","Type":"ContainerDied","Data":"9ee5de18997512b3130b005bf3d1cd9d23e635568eaff535f4da938fb86f1a24"}
Oct 31 04:16:20.511324 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:20.511203    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ee5de18997512b3130b005bf3d1cd9d23e635568eaff535f4da938fb86f1a24"
Oct 31 04:16:36.380556 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:36.380523    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48"]
Oct 31 04:16:36.411069 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:36.411034    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-09c573-qcc48"]
Oct 31 04:16:36.608565 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:16:36.608533    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95f986ba-aa88-4896-9e16-c1701225cb47" path="/var/lib/kubelet/pods/95f986ba-aa88-4896-9e16-c1701225cb47/volumes"
Oct 31 04:16:49.002067 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:16:49.002026    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:17:19.002819 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:17:19.002738    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:17:37.476064 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:17:37.476032    4447 scope.go:122] "RemoveContainer" containerID="7013a88a95e529c6c4121e608800fd45ca7268d6ae1ebf2a8782eb09f241394d"
Oct 31 04:17:37.485784 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:17:37.485766    4447 scope.go:122] "RemoveContainer" containerID="428c623a90a8f78a68ed1b6c30dff1d08607b81e531fe64c2b285d1f35d68aac"
Oct 31 04:17:49.002506 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:17:49.002467    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:18:19.001557 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:18:19.001516    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:18:25.752082 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:18:25.752030    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-lcwvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:18:25.752716 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:18:25.752663    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-lcwvs" podUID="3d7daa66-ba76-4d1a-9de8-2a68928f22b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:18:25.882023 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:18:25.881985    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-dqwsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:18:25.882307 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:18:25.882275    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm" podUID="84553d7f-51d4-4b72-b2dd-ee522e5413ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:18:40.201072 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:18:40.201033    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jgmq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:18:40.201749 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:18:40.201716    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jgmq4" podUID="92da6b1c-2d25-4e59-b877-a9b3b265ab3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:18:49.003179 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:18:49.003093    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:19:19.002168 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:19:19.002127    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:19:35.181635 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:19:35.181608    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 04:19:35.217344 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:19:35.217320    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 04:19:49.001287 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:19:49.001247    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:20:08.878820 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:08.878787    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a058534b-x6p4q"]
Oct 31 04:20:08.879347 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:08.879333    4447 cpu_manager.go:414] "containerMap: removing container" podUID="95f986ba-aa88-4896-9e16-c1701225cb47" containerName="revoke-keys"
Oct 31 04:20:08.879391 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:08.879347    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95f986ba-aa88-4896-9e16-c1701225cb47" containerName="revoke-keys"
Oct 31 04:20:08.879429 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:08.879419    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="95f986ba-aa88-4896-9e16-c1701225cb47" containerName="revoke-keys"
Oct 31 04:20:08.882635 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:08.882618    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a058534b-x6p4q"
Oct 31 04:20:08.938132 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:08.938083    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:20:08.969518 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:08.969478    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf41830-38c9-4891-a570-d57b3ceb050b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a058534b-x6p4q\" (UID: \"2cf41830-38c9-4891-a570-d57b3ceb050b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a058534b-x6p4q"
Oct 31 04:20:08.969710 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:08.969602    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf41830-38c9-4891-a570-d57b3ceb050b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a058534b-x6p4q\" (UID: \"2cf41830-38c9-4891-a570-d57b3ceb050b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a058534b-x6p4q"
Oct 31 04:20:08.969710 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:08.969640    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb26g\" (UniqueName: \"kubernetes.io/projected/2cf41830-38c9-4891-a570-d57b3ceb050b-kube-api-access-wb26g\") pod \"maas-api-revoke-keys-e2e-ait-a058534b-x6p4q\" (UID: \"2cf41830-38c9-4891-a570-d57b3ceb050b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a058534b-x6p4q"
Oct 31 04:20:08.988024 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:08.987996    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tpf5b\""
Oct 31 04:20:08.990841 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:08.990814    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a058534b-x6p4q"]
Oct 31 04:20:09.070558 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:09.070526    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf41830-38c9-4891-a570-d57b3ceb050b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a058534b-x6p4q\" (UID: \"2cf41830-38c9-4891-a570-d57b3ceb050b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a058534b-x6p4q"
Oct 31 04:20:09.070558 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:09.070563    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb26g\" (UniqueName: \"kubernetes.io/projected/2cf41830-38c9-4891-a570-d57b3ceb050b-kube-api-access-wb26g\") pod \"maas-api-revoke-keys-e2e-ait-a058534b-x6p4q\" (UID: \"2cf41830-38c9-4891-a570-d57b3ceb050b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a058534b-x6p4q"
Oct 31 04:20:09.070815 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:09.070670    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf41830-38c9-4891-a570-d57b3ceb050b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a058534b-x6p4q\" (UID: \"2cf41830-38c9-4891-a570-d57b3ceb050b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a058534b-x6p4q"
Oct 31 04:20:09.071207 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:09.071185    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf41830-38c9-4891-a570-d57b3ceb050b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a058534b-x6p4q\" (UID: \"2cf41830-38c9-4891-a570-d57b3ceb050b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a058534b-x6p4q"
Oct 31 04:20:09.073129 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:09.073103    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf41830-38c9-4891-a570-d57b3ceb050b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a058534b-x6p4q\" (UID: \"2cf41830-38c9-4891-a570-d57b3ceb050b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a058534b-x6p4q"
Oct 31 04:20:09.132412 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:09.132344    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb26g\" (UniqueName: \"kubernetes.io/projected/2cf41830-38c9-4891-a570-d57b3ceb050b-kube-api-access-wb26g\") pod \"maas-api-revoke-keys-e2e-ait-a058534b-x6p4q\" (UID: \"2cf41830-38c9-4891-a570-d57b3ceb050b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a058534b-x6p4q"
Oct 31 04:20:09.195693 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:09.195654    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a058534b-x6p4q"
Oct 31 04:20:09.573349 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:09.573323    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a058534b-x6p4q"]
Oct 31 04:20:09.575285 ip-10-0-134-255 kubenswrapper[4447]: W1031 04:20:09.575249    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf41830_38c9_4891_a570_d57b3ceb050b.slice/crio-f118c29e60af6e8355395416687b19e6a788a8ca0f2efd51269e43dd4b6d8f28 WatchSource:0}: Error finding container f118c29e60af6e8355395416687b19e6a788a8ca0f2efd51269e43dd4b6d8f28: Status 404 returned error can't find the container with id f118c29e60af6e8355395416687b19e6a788a8ca0f2efd51269e43dd4b6d8f28
Oct 31 04:20:10.023494 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:10.023461    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a058534b-x6p4q" event={"ID":"2cf41830-38c9-4891-a570-d57b3ceb050b","Type":"ContainerStarted","Data":"b70405c6253fe07017c103b1a6b6ec8f42f447eb8e176ccfec4cc66e11df56d1"}
Oct 31 04:20:10.023900 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:10.023512    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a058534b-x6p4q" event={"ID":"2cf41830-38c9-4891-a570-d57b3ceb050b","Type":"ContainerStarted","Data":"f118c29e60af6e8355395416687b19e6a788a8ca0f2efd51269e43dd4b6d8f28"}
Oct 31 04:20:10.141256 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:10.141151    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a058534b-x6p4q" podStartSLOduration=2.141136981 podStartE2EDuration="2.141136981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:20:10.138980125 +0000 UTC m=+3635.649962506" watchObservedRunningTime="2026-10-31 04:20:10.141136981 +0000 UTC m=+3635.652119358"
Oct 31 04:20:11.039033 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:11.038998    4447 generic.go:356] "Generic (PLEG): container finished" podID="2cf41830-38c9-4891-a570-d57b3ceb050b" containerID="b70405c6253fe07017c103b1a6b6ec8f42f447eb8e176ccfec4cc66e11df56d1" exitCode=0
Oct 31 04:20:11.039484 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:11.039089    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a058534b-x6p4q" event={"ID":"2cf41830-38c9-4891-a570-d57b3ceb050b","Type":"ContainerDied","Data":"b70405c6253fe07017c103b1a6b6ec8f42f447eb8e176ccfec4cc66e11df56d1"}
Oct 31 04:20:12.191714 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:12.191688    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a058534b-x6p4q"
Oct 31 04:20:12.305121 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:12.305041    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cf41830-38c9-4891-a570-d57b3ceb050b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf41830-38c9-4891-a570-d57b3ceb050b-maas-api-openshift-ca\") pod \"2cf41830-38c9-4891-a570-d57b3ceb050b\" (UID: \"2cf41830-38c9-4891-a570-d57b3ceb050b\") "
Oct 31 04:20:12.305281 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:12.305175    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf41830-38c9-4891-a570-d57b3ceb050b-kube-api-access-wb26g\" (UniqueName: \"kubernetes.io/projected/2cf41830-38c9-4891-a570-d57b3ceb050b-kube-api-access-wb26g\") pod \"2cf41830-38c9-4891-a570-d57b3ceb050b\" (UID: \"2cf41830-38c9-4891-a570-d57b3ceb050b\") "
Oct 31 04:20:12.305281 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:12.305221    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cf41830-38c9-4891-a570-d57b3ceb050b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf41830-38c9-4891-a570-d57b3ceb050b-maas-api-xks-ca\") pod \"2cf41830-38c9-4891-a570-d57b3ceb050b\" (UID: \"2cf41830-38c9-4891-a570-d57b3ceb050b\") "
Oct 31 04:20:12.305453 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:12.305426    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cf41830-38c9-4891-a570-d57b3ceb050b-maas-api-openshift-ca" pod "2cf41830-38c9-4891-a570-d57b3ceb050b" (UID: "2cf41830-38c9-4891-a570-d57b3ceb050b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:20:12.307469 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:12.307438    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf41830-38c9-4891-a570-d57b3ceb050b-maas-api-xks-ca" pod "2cf41830-38c9-4891-a570-d57b3ceb050b" (UID: "2cf41830-38c9-4891-a570-d57b3ceb050b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:20:12.307861 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:12.307468    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf41830-38c9-4891-a570-d57b3ceb050b-kube-api-access-wb26g" pod "2cf41830-38c9-4891-a570-d57b3ceb050b" (UID: "2cf41830-38c9-4891-a570-d57b3ceb050b"). InnerVolumeSpecName "kube-api-access-wb26g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:20:12.406972 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:12.406939    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf41830-38c9-4891-a570-d57b3ceb050b-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:20:12.406972 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:12.406967    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb26g\" (UniqueName: \"kubernetes.io/projected/2cf41830-38c9-4891-a570-d57b3ceb050b-kube-api-access-wb26g\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:20:12.407181 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:12.406989    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf41830-38c9-4891-a570-d57b3ceb050b-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:20:13.052154 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:13.052122    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a058534b-x6p4q"
Oct 31 04:20:13.052340 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:13.052162    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a058534b-x6p4q" event={"ID":"2cf41830-38c9-4891-a570-d57b3ceb050b","Type":"ContainerDied","Data":"f118c29e60af6e8355395416687b19e6a788a8ca0f2efd51269e43dd4b6d8f28"}
Oct 31 04:20:13.052340 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:13.052213    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f118c29e60af6e8355395416687b19e6a788a8ca0f2efd51269e43dd4b6d8f28"
Oct 31 04:20:19.002653 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:20:19.002577    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:20:27.614593 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:27.614557    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a058534b-x6p4q"]
Oct 31 04:20:27.664422 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:27.664336    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a058534b-x6p4q"]
Oct 31 04:20:28.607623 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:20:28.607591    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cf41830-38c9-4891-a570-d57b3ceb050b" path="/var/lib/kubelet/pods/2cf41830-38c9-4891-a570-d57b3ceb050b/volumes"
Oct 31 04:20:49.002556 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:20:49.002520    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:21:08.050585 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:21:08.050547    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc4ca437-9shcm"]
Oct 31 04:21:08.100324 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:21:08.100292    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc4ca437-9shcm"]
Oct 31 04:21:08.608235 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:21:08.608200    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a" path="/var/lib/kubelet/pods/c1d40eb9-55c3-41e9-8c89-ba1dbe509c6a/volumes"
Oct 31 04:21:19.002661 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:21:19.002623    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:21:49.003046 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:21:49.002963    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:22:19.002102 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:22:19.002067    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:22:21.721928 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:21.721877    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp"]
Oct 31 04:22:21.722520 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:21.722505    4447 cpu_manager.go:414] "containerMap: removing container" podUID="2cf41830-38c9-4891-a570-d57b3ceb050b" containerName="revoke-keys"
Oct 31 04:22:21.722564 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:21.722524    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf41830-38c9-4891-a570-d57b3ceb050b" containerName="revoke-keys"
Oct 31 04:22:21.722638 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:21.722629    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf41830-38c9-4891-a570-d57b3ceb050b" containerName="revoke-keys"
Oct 31 04:22:21.726041 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:21.726020    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp"
Oct 31 04:22:21.757508 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:21.757473    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c329a83c-4ce8-4c82-8f63-bf8917b3773f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp\" (UID: \"c329a83c-4ce8-4c82-8f63-bf8917b3773f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp"
Oct 31 04:22:21.757508 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:21.757508    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr97r\" (UniqueName: \"kubernetes.io/projected/c329a83c-4ce8-4c82-8f63-bf8917b3773f-kube-api-access-qr97r\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp\" (UID: \"c329a83c-4ce8-4c82-8f63-bf8917b3773f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp"
Oct 31 04:22:21.757769 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:21.757688    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c329a83c-4ce8-4c82-8f63-bf8917b3773f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp\" (UID: \"c329a83c-4ce8-4c82-8f63-bf8917b3773f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp"
Oct 31 04:22:21.784998 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:21.784957    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp"]
Oct 31 04:22:21.829815 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:21.829779    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:22:21.830027 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:21.829932    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tpf5b\""
Oct 31 04:22:21.859091 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:21.859059    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c329a83c-4ce8-4c82-8f63-bf8917b3773f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp\" (UID: \"c329a83c-4ce8-4c82-8f63-bf8917b3773f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp"
Oct 31 04:22:21.859091 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:21.859093    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr97r\" (UniqueName: \"kubernetes.io/projected/c329a83c-4ce8-4c82-8f63-bf8917b3773f-kube-api-access-qr97r\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp\" (UID: \"c329a83c-4ce8-4c82-8f63-bf8917b3773f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp"
Oct 31 04:22:21.859325 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:21.859225    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c329a83c-4ce8-4c82-8f63-bf8917b3773f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp\" (UID: \"c329a83c-4ce8-4c82-8f63-bf8917b3773f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp"
Oct 31 04:22:21.859798 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:21.859779    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c329a83c-4ce8-4c82-8f63-bf8917b3773f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp\" (UID: \"c329a83c-4ce8-4c82-8f63-bf8917b3773f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp"
Oct 31 04:22:21.861527 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:21.861506    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c329a83c-4ce8-4c82-8f63-bf8917b3773f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp\" (UID: \"c329a83c-4ce8-4c82-8f63-bf8917b3773f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp"
Oct 31 04:22:21.920322 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:21.920295    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr97r\" (UniqueName: \"kubernetes.io/projected/c329a83c-4ce8-4c82-8f63-bf8917b3773f-kube-api-access-qr97r\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp\" (UID: \"c329a83c-4ce8-4c82-8f63-bf8917b3773f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp"
Oct 31 04:22:22.038844 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:22.038762    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp"
Oct 31 04:22:22.415963 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:22.415929    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp"]
Oct 31 04:22:22.417549 ip-10-0-134-255 kubenswrapper[4447]: W1031 04:22:22.417526    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc329a83c_4ce8_4c82_8f63_bf8917b3773f.slice/crio-d63f1eba101774532f0cb971f0dd37df78c72046ed95c7c002aeb68cda664b4e WatchSource:0}: Error finding container d63f1eba101774532f0cb971f0dd37df78c72046ed95c7c002aeb68cda664b4e: Status 404 returned error can't find the container with id d63f1eba101774532f0cb971f0dd37df78c72046ed95c7c002aeb68cda664b4e
Oct 31 04:22:22.835839 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:22.835804    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp" event={"ID":"c329a83c-4ce8-4c82-8f63-bf8917b3773f","Type":"ContainerStarted","Data":"8f0321d3b9bc1bbcf5ba04ba1edee65cf83a3e84877d0f2218ca1dcd4cbfe428"}
Oct 31 04:22:22.836273 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:22.835869    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp" event={"ID":"c329a83c-4ce8-4c82-8f63-bf8917b3773f","Type":"ContainerStarted","Data":"d63f1eba101774532f0cb971f0dd37df78c72046ed95c7c002aeb68cda664b4e"}
Oct 31 04:22:22.953661 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:22.953617    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp" podStartSLOduration=1.953603829 podStartE2EDuration="1.953603829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:22:22.951971543 +0000 UTC m=+3768.462953920" watchObservedRunningTime="2026-10-31 04:22:22.953603829 +0000 UTC m=+3768.464586184"
Oct 31 04:22:23.844127 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:23.844088    4447 generic.go:356] "Generic (PLEG): container finished" podID="c329a83c-4ce8-4c82-8f63-bf8917b3773f" containerID="8f0321d3b9bc1bbcf5ba04ba1edee65cf83a3e84877d0f2218ca1dcd4cbfe428" exitCode=0
Oct 31 04:22:23.844522 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:23.844139    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp" event={"ID":"c329a83c-4ce8-4c82-8f63-bf8917b3773f","Type":"ContainerDied","Data":"8f0321d3b9bc1bbcf5ba04ba1edee65cf83a3e84877d0f2218ca1dcd4cbfe428"}
Oct 31 04:22:24.993123 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:24.993099    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp"
Oct 31 04:22:25.094253 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:25.094218    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c329a83c-4ce8-4c82-8f63-bf8917b3773f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c329a83c-4ce8-4c82-8f63-bf8917b3773f-maas-api-openshift-ca\") pod \"c329a83c-4ce8-4c82-8f63-bf8917b3773f\" (UID: \"c329a83c-4ce8-4c82-8f63-bf8917b3773f\") "
Oct 31 04:22:25.094430 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:25.094318    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c329a83c-4ce8-4c82-8f63-bf8917b3773f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c329a83c-4ce8-4c82-8f63-bf8917b3773f-maas-api-xks-ca\") pod \"c329a83c-4ce8-4c82-8f63-bf8917b3773f\" (UID: \"c329a83c-4ce8-4c82-8f63-bf8917b3773f\") "
Oct 31 04:22:25.094430 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:25.094394    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c329a83c-4ce8-4c82-8f63-bf8917b3773f-kube-api-access-qr97r\" (UniqueName: \"kubernetes.io/projected/c329a83c-4ce8-4c82-8f63-bf8917b3773f-kube-api-access-qr97r\") pod \"c329a83c-4ce8-4c82-8f63-bf8917b3773f\" (UID: \"c329a83c-4ce8-4c82-8f63-bf8917b3773f\") "
Oct 31 04:22:25.094655 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:25.094633    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c329a83c-4ce8-4c82-8f63-bf8917b3773f-maas-api-openshift-ca" pod "c329a83c-4ce8-4c82-8f63-bf8917b3773f" (UID: "c329a83c-4ce8-4c82-8f63-bf8917b3773f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:22:25.094863 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:25.094837    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c329a83c-4ce8-4c82-8f63-bf8917b3773f-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:22:25.096663 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:25.096641    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c329a83c-4ce8-4c82-8f63-bf8917b3773f-maas-api-xks-ca" pod "c329a83c-4ce8-4c82-8f63-bf8917b3773f" (UID: "c329a83c-4ce8-4c82-8f63-bf8917b3773f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:22:25.096742 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:25.096642    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c329a83c-4ce8-4c82-8f63-bf8917b3773f-kube-api-access-qr97r" pod "c329a83c-4ce8-4c82-8f63-bf8917b3773f" (UID: "c329a83c-4ce8-4c82-8f63-bf8917b3773f"). InnerVolumeSpecName "kube-api-access-qr97r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:22:25.195852 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:25.195818    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c329a83c-4ce8-4c82-8f63-bf8917b3773f-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:22:25.195852 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:25.195848    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr97r\" (UniqueName: \"kubernetes.io/projected/c329a83c-4ce8-4c82-8f63-bf8917b3773f-kube-api-access-qr97r\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:22:25.857304 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:25.857275    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp"
Oct 31 04:22:25.857490 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:25.857316    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp" event={"ID":"c329a83c-4ce8-4c82-8f63-bf8917b3773f","Type":"ContainerDied","Data":"d63f1eba101774532f0cb971f0dd37df78c72046ed95c7c002aeb68cda664b4e"}
Oct 31 04:22:25.857490 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:25.857361    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d63f1eba101774532f0cb971f0dd37df78c72046ed95c7c002aeb68cda664b4e"
Oct 31 04:22:37.552535 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:37.552498    4447 scope.go:122] "RemoveContainer" containerID="536c8b472579513f79bcee66e57e3f9963b470e0b12fe4a71cfa40c6de2b4aa0"
Oct 31 04:22:37.561922 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:37.561864    4447 scope.go:122] "RemoveContainer" containerID="9fefcc9f0150ad5d0941600f915a828dfe01756bd0d55787cb8ae926a7ffec81"
Oct 31 04:22:46.696913 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:46.696866    4447 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp"]
Oct 31 04:22:46.746666 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:46.746621    4447 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0bda5d-wrjmp"]
Oct 31 04:22:48.607445 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:22:48.607410    4447 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c329a83c-4ce8-4c82-8f63-bf8917b3773f" path="/var/lib/kubelet/pods/c329a83c-4ce8-4c82-8f63-bf8917b3773f/volumes"
Oct 31 04:22:49.002009 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:22:49.001975    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:23:19.002533 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:23:19.002457    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:23:21.012819 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:21.012782    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8ptzk"]
Oct 31 04:23:21.013517 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:21.013494    4447 cpu_manager.go:414] "containerMap: removing container" podUID="c329a83c-4ce8-4c82-8f63-bf8917b3773f" containerName="revoke-keys"
Oct 31 04:23:21.013617 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:21.013520    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c329a83c-4ce8-4c82-8f63-bf8917b3773f" containerName="revoke-keys"
Oct 31 04:23:21.013675 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:21.013664    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="c329a83c-4ce8-4c82-8f63-bf8917b3773f" containerName="revoke-keys"
Oct 31 04:23:21.017055 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:21.017035    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8ptzk"
Oct 31 04:23:21.120175 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:21.120143    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tpf5b\""
Oct 31 04:23:21.120175 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:21.120166    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:23:21.122912 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:21.122879    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8ptzk"]
Oct 31 04:23:21.181735 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:21.181701    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq9fv\" (UniqueName: \"kubernetes.io/projected/932391ee-8722-4469-a7f3-cfbd4a67e8ae-kube-api-access-bq9fv\") pod \"maas-api-revoke-sub-f03629db-8ptzk\" (UID: \"932391ee-8722-4469-a7f3-cfbd4a67e8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8ptzk"
Oct 31 04:23:21.181982 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:21.181879    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/932391ee-8722-4469-a7f3-cfbd4a67e8ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-8ptzk\" (UID: \"932391ee-8722-4469-a7f3-cfbd4a67e8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8ptzk"
Oct 31 04:23:21.182077 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:21.181981    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/932391ee-8722-4469-a7f3-cfbd4a67e8ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-8ptzk\" (UID: \"932391ee-8722-4469-a7f3-cfbd4a67e8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8ptzk"
Oct 31 04:23:21.282719 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:21.282614    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/932391ee-8722-4469-a7f3-cfbd4a67e8ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-8ptzk\" (UID: \"932391ee-8722-4469-a7f3-cfbd4a67e8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8ptzk"
Oct 31 04:23:21.282719 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:21.282665    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/932391ee-8722-4469-a7f3-cfbd4a67e8ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-8ptzk\" (UID: \"932391ee-8722-4469-a7f3-cfbd4a67e8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8ptzk"
Oct 31 04:23:21.282719 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:21.282717    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq9fv\" (UniqueName: \"kubernetes.io/projected/932391ee-8722-4469-a7f3-cfbd4a67e8ae-kube-api-access-bq9fv\") pod \"maas-api-revoke-sub-f03629db-8ptzk\" (UID: \"932391ee-8722-4469-a7f3-cfbd4a67e8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8ptzk"
Oct 31 04:23:21.283454 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:21.283427    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/932391ee-8722-4469-a7f3-cfbd4a67e8ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-8ptzk\" (UID: \"932391ee-8722-4469-a7f3-cfbd4a67e8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8ptzk"
Oct 31 04:23:21.285247 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:21.285223    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/932391ee-8722-4469-a7f3-cfbd4a67e8ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-8ptzk\" (UID: \"932391ee-8722-4469-a7f3-cfbd4a67e8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8ptzk"
Oct 31 04:23:21.349230 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:21.349200    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq9fv\" (UniqueName: \"kubernetes.io/projected/932391ee-8722-4469-a7f3-cfbd4a67e8ae-kube-api-access-bq9fv\") pod \"maas-api-revoke-sub-f03629db-8ptzk\" (UID: \"932391ee-8722-4469-a7f3-cfbd4a67e8ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8ptzk"
Oct 31 04:23:21.629407 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:21.629309    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8ptzk"
Oct 31 04:23:22.006630 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:22.006601    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8ptzk"]
Oct 31 04:23:22.008354 ip-10-0-134-255 kubenswrapper[4447]: W1031 04:23:22.008324    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932391ee_8722_4469_a7f3_cfbd4a67e8ae.slice/crio-20b2d4bdac0ccbb0888aaf559f00f5697d4326e656359302710ae35ab382d3f4 WatchSource:0}: Error finding container 20b2d4bdac0ccbb0888aaf559f00f5697d4326e656359302710ae35ab382d3f4: Status 404 returned error can't find the container with id 20b2d4bdac0ccbb0888aaf559f00f5697d4326e656359302710ae35ab382d3f4
Oct 31 04:23:22.188004 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:22.187972    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8ptzk" event={"ID":"932391ee-8722-4469-a7f3-cfbd4a67e8ae","Type":"ContainerStarted","Data":"20b2d4bdac0ccbb0888aaf559f00f5697d4326e656359302710ae35ab382d3f4"}
Oct 31 04:23:23.196105 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:23.196013    4447 generic.go:356] "Generic (PLEG): container finished" podID="932391ee-8722-4469-a7f3-cfbd4a67e8ae" containerID="a677dd5fb77451fcc0eab3aa1b77aac78e9c6a524abcd3b9be0954584a0f86ea" exitCode=0
Oct 31 04:23:23.196501 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:23.196100    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8ptzk" event={"ID":"932391ee-8722-4469-a7f3-cfbd4a67e8ae","Type":"ContainerDied","Data":"a677dd5fb77451fcc0eab3aa1b77aac78e9c6a524abcd3b9be0954584a0f86ea"}
Oct 31 04:23:24.348433 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:24.348406    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8ptzk"
Oct 31 04:23:24.516756 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:24.516721    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/932391ee-8722-4469-a7f3-cfbd4a67e8ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/932391ee-8722-4469-a7f3-cfbd4a67e8ae-maas-api-openshift-ca\") pod \"932391ee-8722-4469-a7f3-cfbd4a67e8ae\" (UID: \"932391ee-8722-4469-a7f3-cfbd4a67e8ae\") "
Oct 31 04:23:24.516990 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:24.516775    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/932391ee-8722-4469-a7f3-cfbd4a67e8ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/932391ee-8722-4469-a7f3-cfbd4a67e8ae-maas-api-xks-ca\") pod \"932391ee-8722-4469-a7f3-cfbd4a67e8ae\" (UID: \"932391ee-8722-4469-a7f3-cfbd4a67e8ae\") "
Oct 31 04:23:24.516990 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:24.516873    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/932391ee-8722-4469-a7f3-cfbd4a67e8ae-kube-api-access-bq9fv\" (UniqueName: \"kubernetes.io/projected/932391ee-8722-4469-a7f3-cfbd4a67e8ae-kube-api-access-bq9fv\") pod \"932391ee-8722-4469-a7f3-cfbd4a67e8ae\" (UID: \"932391ee-8722-4469-a7f3-cfbd4a67e8ae\") "
Oct 31 04:23:24.517182 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:24.517155    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932391ee-8722-4469-a7f3-cfbd4a67e8ae-maas-api-openshift-ca" pod "932391ee-8722-4469-a7f3-cfbd4a67e8ae" (UID: "932391ee-8722-4469-a7f3-cfbd4a67e8ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:23:24.519040 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:24.519016    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/932391ee-8722-4469-a7f3-cfbd4a67e8ae-maas-api-xks-ca" pod "932391ee-8722-4469-a7f3-cfbd4a67e8ae" (UID: "932391ee-8722-4469-a7f3-cfbd4a67e8ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:23:24.519149 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:24.519129    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932391ee-8722-4469-a7f3-cfbd4a67e8ae-kube-api-access-bq9fv" pod "932391ee-8722-4469-a7f3-cfbd4a67e8ae" (UID: "932391ee-8722-4469-a7f3-cfbd4a67e8ae"). InnerVolumeSpecName "kube-api-access-bq9fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:23:24.618219 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:24.618192    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/932391ee-8722-4469-a7f3-cfbd4a67e8ae-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:23:24.618219 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:24.618218    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/932391ee-8722-4469-a7f3-cfbd4a67e8ae-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:23:24.618375 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:24.618228    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq9fv\" (UniqueName: \"kubernetes.io/projected/932391ee-8722-4469-a7f3-cfbd4a67e8ae-kube-api-access-bq9fv\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:23:25.210258 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:25.210227    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8ptzk"
Oct 31 04:23:25.210428 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:25.210265    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8ptzk" event={"ID":"932391ee-8722-4469-a7f3-cfbd4a67e8ae","Type":"ContainerDied","Data":"20b2d4bdac0ccbb0888aaf559f00f5697d4326e656359302710ae35ab382d3f4"}
Oct 31 04:23:25.210428 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:25.210322    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20b2d4bdac0ccbb0888aaf559f00f5697d4326e656359302710ae35ab382d3f4"
Oct 31 04:23:40.872071 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:40.872022    4447 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-dqwsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:23:40.872587 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:40.872110    4447 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-dqwsm" podUID="84553d7f-51d4-4b72-b2dd-ee522e5413ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:23:41.825749 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:41.825709    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w4hwp"]
Oct 31 04:23:41.826310 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:41.826294    4447 cpu_manager.go:414] "containerMap: removing container" podUID="932391ee-8722-4469-a7f3-cfbd4a67e8ae" containerName="revoke-keys"
Oct 31 04:23:41.826398 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:41.826311    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="932391ee-8722-4469-a7f3-cfbd4a67e8ae" containerName="revoke-keys"
Oct 31 04:23:41.826398 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:41.826394    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="932391ee-8722-4469-a7f3-cfbd4a67e8ae" containerName="revoke-keys"
Oct 31 04:23:41.829642 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:41.829628    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w4hwp"
Oct 31 04:23:41.888458 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:41.888431    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w4hwp"]
Oct 31 04:23:41.934662 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:41.934627    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:23:41.934844 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:41.934734    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tpf5b\""
Oct 31 04:23:41.953982 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:41.953953    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr57j\" (UniqueName: \"kubernetes.io/projected/cfc459fa-892a-46ae-91f6-cf0c1a7acf7d-kube-api-access-tr57j\") pod \"maas-api-revoke-sub-0165d7d2-w4hwp\" (UID: \"cfc459fa-892a-46ae-91f6-cf0c1a7acf7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w4hwp"
Oct 31 04:23:41.954181 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:41.954152    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc459fa-892a-46ae-91f6-cf0c1a7acf7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-w4hwp\" (UID: \"cfc459fa-892a-46ae-91f6-cf0c1a7acf7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w4hwp"
Oct 31 04:23:41.954246 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:41.954193    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc459fa-892a-46ae-91f6-cf0c1a7acf7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-w4hwp\" (UID: \"cfc459fa-892a-46ae-91f6-cf0c1a7acf7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w4hwp"
Oct 31 04:23:42.055663 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:42.055626    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc459fa-892a-46ae-91f6-cf0c1a7acf7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-w4hwp\" (UID: \"cfc459fa-892a-46ae-91f6-cf0c1a7acf7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w4hwp"
Oct 31 04:23:42.055846 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:42.055694    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr57j\" (UniqueName: \"kubernetes.io/projected/cfc459fa-892a-46ae-91f6-cf0c1a7acf7d-kube-api-access-tr57j\") pod \"maas-api-revoke-sub-0165d7d2-w4hwp\" (UID: \"cfc459fa-892a-46ae-91f6-cf0c1a7acf7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w4hwp"
Oct 31 04:23:42.055846 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:42.055799    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc459fa-892a-46ae-91f6-cf0c1a7acf7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-w4hwp\" (UID: \"cfc459fa-892a-46ae-91f6-cf0c1a7acf7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w4hwp"
Oct 31 04:23:42.056402 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:42.056383    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc459fa-892a-46ae-91f6-cf0c1a7acf7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-w4hwp\" (UID: \"cfc459fa-892a-46ae-91f6-cf0c1a7acf7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w4hwp"
Oct 31 04:23:42.058084 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:42.058066    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc459fa-892a-46ae-91f6-cf0c1a7acf7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-w4hwp\" (UID: \"cfc459fa-892a-46ae-91f6-cf0c1a7acf7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w4hwp"
Oct 31 04:23:42.117620 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:42.117535    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr57j\" (UniqueName: \"kubernetes.io/projected/cfc459fa-892a-46ae-91f6-cf0c1a7acf7d-kube-api-access-tr57j\") pod \"maas-api-revoke-sub-0165d7d2-w4hwp\" (UID: \"cfc459fa-892a-46ae-91f6-cf0c1a7acf7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w4hwp"
Oct 31 04:23:42.143296 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:42.143265    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w4hwp"
Oct 31 04:23:42.518474 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:42.518445    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w4hwp"]
Oct 31 04:23:42.520309 ip-10-0-134-255 kubenswrapper[4447]: W1031 04:23:42.520276    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc459fa_892a_46ae_91f6_cf0c1a7acf7d.slice/crio-0edb9b22fe02cde917fa6f743f78fb7869ddd8342d23a210c30cd1c00937d173 WatchSource:0}: Error finding container 0edb9b22fe02cde917fa6f743f78fb7869ddd8342d23a210c30cd1c00937d173: Status 404 returned error can't find the container with id 0edb9b22fe02cde917fa6f743f78fb7869ddd8342d23a210c30cd1c00937d173
Oct 31 04:23:43.357361 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:43.357325    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w4hwp" event={"ID":"cfc459fa-892a-46ae-91f6-cf0c1a7acf7d","Type":"ContainerStarted","Data":"636f9ae177f2ea06766db58c02cb956ad99d2d2c67390e31da64e609483f3688"}
Oct 31 04:23:43.357786 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:43.357381    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w4hwp" event={"ID":"cfc459fa-892a-46ae-91f6-cf0c1a7acf7d","Type":"ContainerStarted","Data":"0edb9b22fe02cde917fa6f743f78fb7869ddd8342d23a210c30cd1c00937d173"}
Oct 31 04:23:43.475462 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:43.475407    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w4hwp" podStartSLOduration=2.475394277 podStartE2EDuration="2.475394277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:43.473254139 +0000 UTC m=+3848.984236517" watchObservedRunningTime="2026-10-31 04:23:43.475394277 +0000 UTC m=+3848.986376704"
Oct 31 04:23:44.364361 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:44.364325    4447 generic.go:356] "Generic (PLEG): container finished" podID="cfc459fa-892a-46ae-91f6-cf0c1a7acf7d" containerID="636f9ae177f2ea06766db58c02cb956ad99d2d2c67390e31da64e609483f3688" exitCode=0
Oct 31 04:23:44.364818 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:44.364407    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w4hwp" event={"ID":"cfc459fa-892a-46ae-91f6-cf0c1a7acf7d","Type":"ContainerDied","Data":"636f9ae177f2ea06766db58c02cb956ad99d2d2c67390e31da64e609483f3688"}
Oct 31 04:23:45.515510 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:45.515483    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w4hwp"
Oct 31 04:23:45.593081 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:45.593051    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc459fa-892a-46ae-91f6-cf0c1a7acf7d-kube-api-access-tr57j\" (UniqueName: \"kubernetes.io/projected/cfc459fa-892a-46ae-91f6-cf0c1a7acf7d-kube-api-access-tr57j\") pod \"cfc459fa-892a-46ae-91f6-cf0c1a7acf7d\" (UID: \"cfc459fa-892a-46ae-91f6-cf0c1a7acf7d\") "
Oct 31 04:23:45.593259 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:45.593128    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfc459fa-892a-46ae-91f6-cf0c1a7acf7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc459fa-892a-46ae-91f6-cf0c1a7acf7d-maas-api-xks-ca\") pod \"cfc459fa-892a-46ae-91f6-cf0c1a7acf7d\" (UID: \"cfc459fa-892a-46ae-91f6-cf0c1a7acf7d\") "
Oct 31 04:23:45.593259 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:45.593161    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfc459fa-892a-46ae-91f6-cf0c1a7acf7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc459fa-892a-46ae-91f6-cf0c1a7acf7d-maas-api-openshift-ca\") pod \"cfc459fa-892a-46ae-91f6-cf0c1a7acf7d\" (UID: \"cfc459fa-892a-46ae-91f6-cf0c1a7acf7d\") "
Oct 31 04:23:45.593564 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:45.593541    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc459fa-892a-46ae-91f6-cf0c1a7acf7d-maas-api-openshift-ca" pod "cfc459fa-892a-46ae-91f6-cf0c1a7acf7d" (UID: "cfc459fa-892a-46ae-91f6-cf0c1a7acf7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:23:45.595267 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:45.595243    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfc459fa-892a-46ae-91f6-cf0c1a7acf7d-maas-api-xks-ca" pod "cfc459fa-892a-46ae-91f6-cf0c1a7acf7d" (UID: "cfc459fa-892a-46ae-91f6-cf0c1a7acf7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:23:45.595371 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:45.595265    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc459fa-892a-46ae-91f6-cf0c1a7acf7d-kube-api-access-tr57j" pod "cfc459fa-892a-46ae-91f6-cf0c1a7acf7d" (UID: "cfc459fa-892a-46ae-91f6-cf0c1a7acf7d"). InnerVolumeSpecName "kube-api-access-tr57j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:23:45.694080 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:45.693991    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr57j\" (UniqueName: \"kubernetes.io/projected/cfc459fa-892a-46ae-91f6-cf0c1a7acf7d-kube-api-access-tr57j\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:23:45.694080 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:45.694022    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc459fa-892a-46ae-91f6-cf0c1a7acf7d-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:23:45.694080 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:45.694037    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc459fa-892a-46ae-91f6-cf0c1a7acf7d-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:23:46.378591 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:46.378557    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w4hwp" event={"ID":"cfc459fa-892a-46ae-91f6-cf0c1a7acf7d","Type":"ContainerDied","Data":"0edb9b22fe02cde917fa6f743f78fb7869ddd8342d23a210c30cd1c00937d173"}
Oct 31 04:23:46.378591 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:46.378582    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w4hwp"
Oct 31 04:23:46.378818 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:46.378614    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0edb9b22fe02cde917fa6f743f78fb7869ddd8342d23a210c30cd1c00937d173"
Oct 31 04:23:49.003153 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:23:49.003117    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:23:53.642237 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:53.642201    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vj66p"]
Oct 31 04:23:53.642810 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:53.642795    4447 cpu_manager.go:414] "containerMap: removing container" podUID="cfc459fa-892a-46ae-91f6-cf0c1a7acf7d" containerName="revoke-keys"
Oct 31 04:23:53.642854 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:53.642811    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc459fa-892a-46ae-91f6-cf0c1a7acf7d" containerName="revoke-keys"
Oct 31 04:23:53.642910 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:53.642901    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc459fa-892a-46ae-91f6-cf0c1a7acf7d" containerName="revoke-keys"
Oct 31 04:23:53.646319 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:53.646304    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vj66p"
Oct 31 04:23:53.682204 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:53.682167    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db7f8389-426b-4d60-ab96-a08a3cbbd391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-vj66p\" (UID: \"db7f8389-426b-4d60-ab96-a08a3cbbd391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vj66p"
Oct 31 04:23:53.682416 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:53.682366    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db7f8389-426b-4d60-ab96-a08a3cbbd391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-vj66p\" (UID: \"db7f8389-426b-4d60-ab96-a08a3cbbd391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vj66p"
Oct 31 04:23:53.682628 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:53.682608    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmf7l\" (UniqueName: \"kubernetes.io/projected/db7f8389-426b-4d60-ab96-a08a3cbbd391-kube-api-access-cmf7l\") pod \"maas-api-revoke-sub-6b19338f-vj66p\" (UID: \"db7f8389-426b-4d60-ab96-a08a3cbbd391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vj66p"
Oct 31 04:23:53.704259 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:53.704221    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vj66p"]
Oct 31 04:23:53.750178 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:53.750139    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:23:53.750178 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:53.750163    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tpf5b\""
Oct 31 04:23:53.783446 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:53.783402    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmf7l\" (UniqueName: \"kubernetes.io/projected/db7f8389-426b-4d60-ab96-a08a3cbbd391-kube-api-access-cmf7l\") pod \"maas-api-revoke-sub-6b19338f-vj66p\" (UID: \"db7f8389-426b-4d60-ab96-a08a3cbbd391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vj66p"
Oct 31 04:23:53.783671 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:53.783512    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db7f8389-426b-4d60-ab96-a08a3cbbd391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-vj66p\" (UID: \"db7f8389-426b-4d60-ab96-a08a3cbbd391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vj66p"
Oct 31 04:23:53.783671 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:53.783586    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db7f8389-426b-4d60-ab96-a08a3cbbd391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-vj66p\" (UID: \"db7f8389-426b-4d60-ab96-a08a3cbbd391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vj66p"
Oct 31 04:23:53.784409 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:53.784376    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db7f8389-426b-4d60-ab96-a08a3cbbd391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-vj66p\" (UID: \"db7f8389-426b-4d60-ab96-a08a3cbbd391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vj66p"
Oct 31 04:23:53.785928 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:53.785904    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db7f8389-426b-4d60-ab96-a08a3cbbd391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-vj66p\" (UID: \"db7f8389-426b-4d60-ab96-a08a3cbbd391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vj66p"
Oct 31 04:23:53.845051 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:53.845022    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmf7l\" (UniqueName: \"kubernetes.io/projected/db7f8389-426b-4d60-ab96-a08a3cbbd391-kube-api-access-cmf7l\") pod \"maas-api-revoke-sub-6b19338f-vj66p\" (UID: \"db7f8389-426b-4d60-ab96-a08a3cbbd391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vj66p"
Oct 31 04:23:53.959971 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:53.959936    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vj66p"
Oct 31 04:23:54.343951 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:54.343922    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vj66p"]
Oct 31 04:23:54.345233 ip-10-0-134-255 kubenswrapper[4447]: W1031 04:23:54.345207    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb7f8389_426b_4d60_ab96_a08a3cbbd391.slice/crio-42abacef3d4936798a677ef7063166593d7db2b447f41bd4a5c99b74923c20b6 WatchSource:0}: Error finding container 42abacef3d4936798a677ef7063166593d7db2b447f41bd4a5c99b74923c20b6: Status 404 returned error can't find the container with id 42abacef3d4936798a677ef7063166593d7db2b447f41bd4a5c99b74923c20b6
Oct 31 04:23:54.427503 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:54.427472    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vj66p" event={"ID":"db7f8389-426b-4d60-ab96-a08a3cbbd391","Type":"ContainerStarted","Data":"42abacef3d4936798a677ef7063166593d7db2b447f41bd4a5c99b74923c20b6"}
Oct 31 04:23:55.435272 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:55.435238    4447 generic.go:356] "Generic (PLEG): container finished" podID="db7f8389-426b-4d60-ab96-a08a3cbbd391" containerID="0122730d05573148505dbdafd6ba8c1d3801a1d76e66641055ad3fe152f4a9af" exitCode=0
Oct 31 04:23:55.435685 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:55.435332    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vj66p" event={"ID":"db7f8389-426b-4d60-ab96-a08a3cbbd391","Type":"ContainerDied","Data":"0122730d05573148505dbdafd6ba8c1d3801a1d76e66641055ad3fe152f4a9af"}
Oct 31 04:23:56.588935 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:56.588879    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vj66p"
Oct 31 04:23:56.717772 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:56.717742    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db7f8389-426b-4d60-ab96-a08a3cbbd391-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db7f8389-426b-4d60-ab96-a08a3cbbd391-maas-api-xks-ca\") pod \"db7f8389-426b-4d60-ab96-a08a3cbbd391\" (UID: \"db7f8389-426b-4d60-ab96-a08a3cbbd391\") "
Oct 31 04:23:56.717988 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:56.717815    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db7f8389-426b-4d60-ab96-a08a3cbbd391-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db7f8389-426b-4d60-ab96-a08a3cbbd391-maas-api-openshift-ca\") pod \"db7f8389-426b-4d60-ab96-a08a3cbbd391\" (UID: \"db7f8389-426b-4d60-ab96-a08a3cbbd391\") "
Oct 31 04:23:56.717988 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:56.717976    4447 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db7f8389-426b-4d60-ab96-a08a3cbbd391-kube-api-access-cmf7l\" (UniqueName: \"kubernetes.io/projected/db7f8389-426b-4d60-ab96-a08a3cbbd391-kube-api-access-cmf7l\") pod \"db7f8389-426b-4d60-ab96-a08a3cbbd391\" (UID: \"db7f8389-426b-4d60-ab96-a08a3cbbd391\") "
Oct 31 04:23:56.718257 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:56.718228    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db7f8389-426b-4d60-ab96-a08a3cbbd391-maas-api-openshift-ca" pod "db7f8389-426b-4d60-ab96-a08a3cbbd391" (UID: "db7f8389-426b-4d60-ab96-a08a3cbbd391"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:23:56.720093 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:56.720060    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db7f8389-426b-4d60-ab96-a08a3cbbd391-maas-api-xks-ca" pod "db7f8389-426b-4d60-ab96-a08a3cbbd391" (UID: "db7f8389-426b-4d60-ab96-a08a3cbbd391"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:23:56.720279 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:56.720258    4447 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db7f8389-426b-4d60-ab96-a08a3cbbd391-kube-api-access-cmf7l" pod "db7f8389-426b-4d60-ab96-a08a3cbbd391" (UID: "db7f8389-426b-4d60-ab96-a08a3cbbd391"). InnerVolumeSpecName "kube-api-access-cmf7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:23:56.818893 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:56.818859    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db7f8389-426b-4d60-ab96-a08a3cbbd391-maas-api-openshift-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:23:56.819093 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:56.818907    4447 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmf7l\" (UniqueName: \"kubernetes.io/projected/db7f8389-426b-4d60-ab96-a08a3cbbd391-kube-api-access-cmf7l\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:23:56.819093 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:56.818923    4447 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db7f8389-426b-4d60-ab96-a08a3cbbd391-maas-api-xks-ca\") on node \"ip-10-0-134-255.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 04:23:57.448842 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:57.448810    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vj66p" event={"ID":"db7f8389-426b-4d60-ab96-a08a3cbbd391","Type":"ContainerDied","Data":"42abacef3d4936798a677ef7063166593d7db2b447f41bd4a5c99b74923c20b6"}
Oct 31 04:23:57.448842 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:57.448828    4447 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vj66p"
Oct 31 04:23:57.449082 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:23:57.448862    4447 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42abacef3d4936798a677ef7063166593d7db2b447f41bd4a5c99b74923c20b6"
Oct 31 04:24:16.950132 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:16.950103    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-66b48bb959-2vdzd_1267ec47-4190-45eb-bd44-2850e7af98e0/manager/0.log"
Oct 31 04:24:17.322923 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:17.322895    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-cbh4k_da0b818f-0cfd-4c31-a808-588b3138c91b/manager/0.log"
Oct 31 04:24:17.556356 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:17.556319    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8ws7p_90781578-9f71-4512-8141-d08c8830483b/manager/0.log"
Oct 31 04:24:18.662433 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:18.662407    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vvkxg_c9241326-bbbe-4a12-aabd-67e56ee8e55e/server/0.log"
Oct 31 04:24:18.900783 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:18.900729    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-fh6m6_73b2caea-8bec-45ab-ad59-0415870d7b05/manager/0.log"
Oct 31 04:24:19.002872 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:24:19.002832    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:24:19.167356 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:19.167327    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-q5brx_23ab1790-f42f-4bad-a72f-49800dee980c/proxy/0.log"
Oct 31 04:24:20.521361 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:20.521332    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-w4hwp_cfc459fa-892a-46ae-91f6-cf0c1a7acf7d/revoke-keys/0.log"
Oct 31 04:24:20.744412 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:20.744383    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-vj66p_db7f8389-426b-4d60-ab96-a08a3cbbd391/revoke-keys/0.log"
Oct 31 04:24:20.972230 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:20.972202    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-8ptzk_932391ee-8722-4469-a7f3-cfbd4a67e8ae/revoke-keys/0.log"
Oct 31 04:24:21.199798 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:21.199773    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-nnpjm_290bb433-86e4-4bcb-b609-abb84f34705f/postgres/0.log"
Oct 31 04:24:23.187587 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:23.187557    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-4bnf2_9670f62f-e977-4370-951a-2a7bcb9209ac/manager/0.log"
Oct 31 04:24:23.512382 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:23.512350    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-mtv6q_f4fc7df3-3489-458c-971e-7f2717626680/manager/0.log"
Oct 31 04:24:23.732744 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:23.732714    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vhpht_dd0803af-c8ee-4ae0-8082-1c4e47742909/kuadrant-console-plugin/0.log"
Oct 31 04:24:24.636643 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:24.636610    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-275tg_a5ebe756-cf88-441b-aba4-51bb5f9d1922/manager/0.log"
Oct 31 04:24:25.438534 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:25.438500    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7_c5de1f7a-3665-4f3c-9f23-3febe52c88d0/istio-proxy/0.log"
Oct 31 04:24:25.664273 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:25.664242    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-cx854_70b3d31c-a1ec-45db-97e5-cf9d1a0f227c/discovery/0.log"
Oct 31 04:24:26.630177 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:26.630145    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-dqwsm_84553d7f-51d4-4b72-b2dd-ee522e5413ad/istio-proxy/0.log"
Oct 31 04:24:26.863141 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:26.863115    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-jgmq4_92da6b1c-2d25-4e59-b877-a9b3b265ab3f/istio-proxy/0.log"
Oct 31 04:24:27.604292 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:27.604255    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-lcwvs_3d7daa66-ba76-4d1a-9de8-2a68928f22b7/istio-proxy/0.log"
Oct 31 04:24:28.844634 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:28.844559    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v_c539e6bf-2bd3-482c-9881-5c3fd79143e2/istio-proxy/0.log"
Oct 31 04:24:29.301077 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:29.301043    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-2z9q8_40e3764e-c5f1-4c75-b49e-13d815a622d2/payload-processing/0.log"
Oct 31 04:24:30.372222 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:30.372191    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb_2b7caa8c-a0aa-4fbd-8918-288e39fc4f56/main/0.log"
Oct 31 04:24:30.644467 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:30.644386    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-nbrhn_98f65062-e7db-4f54-9d76-6addebbebae6/main/0.log"
Oct 31 04:24:30.878184 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:30.878152    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-xl2pv_a3767bbe-f08f-47b5-84eb-f01ebf697a41/main/0.log"
Oct 31 04:24:31.100747 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:31.100718    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-dsdwd_be4ef64f-41b3-4ec7-9408-bbbb03f85575/main/0.log"
Oct 31 04:24:31.317267 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:31.317235    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r_7523be11-f92d-47cc-9309-998e5e4caa6b/main/0.log"
Oct 31 04:24:31.532574 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:31.532552    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-s6nfq_41c075e6-b57b-4e72-ad58-59450e64c8a3/main/0.log"
Oct 31 04:24:31.756325 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:31.756297    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5_5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7/main/0.log"
Oct 31 04:24:35.222030 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:35.222000    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 04:24:35.260026 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:35.259998    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkkr4_7d72ff8d-4480-4266-ab8b-a225f4245836/ovn-acl-logging/0.log"
Oct 31 04:24:42.334152 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:42.334103    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-66b48bb959-2vdzd_1267ec47-4190-45eb-bd44-2850e7af98e0/manager/0.log"
Oct 31 04:24:42.703389 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:42.703282    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-cbh4k_da0b818f-0cfd-4c31-a808-588b3138c91b/manager/0.log"
Oct 31 04:24:42.937449 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:42.937408    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8ws7p_90781578-9f71-4512-8141-d08c8830483b/manager/0.log"
Oct 31 04:24:43.758592 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:43.758497    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vvkxg_c9241326-bbbe-4a12-aabd-67e56ee8e55e/server/0.log"
Oct 31 04:24:43.992313 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:43.992274    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-fh6m6_73b2caea-8bec-45ab-ad59-0415870d7b05/manager/0.log"
Oct 31 04:24:44.256243 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:44.256204    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-q5brx_23ab1790-f42f-4bad-a72f-49800dee980c/proxy/0.log"
Oct 31 04:24:45.415976 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:45.415946    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-w4hwp_cfc459fa-892a-46ae-91f6-cf0c1a7acf7d/revoke-keys/0.log"
Oct 31 04:24:45.646642 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:45.646615    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-vj66p_db7f8389-426b-4d60-ab96-a08a3cbbd391/revoke-keys/0.log"
Oct 31 04:24:45.879594 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:45.879565    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-8ptzk_932391ee-8722-4469-a7f3-cfbd4a67e8ae/revoke-keys/0.log"
Oct 31 04:24:46.099266 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:46.099232    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-nnpjm_290bb433-86e4-4bcb-b609-abb84f34705f/postgres/0.log"
Oct 31 04:24:47.804154 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:47.804082    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-4bnf2_9670f62f-e977-4370-951a-2a7bcb9209ac/manager/0.log"
Oct 31 04:24:48.126905 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:48.126813    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-mtv6q_f4fc7df3-3489-458c-971e-7f2717626680/manager/0.log"
Oct 31 04:24:48.359465 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:48.359437    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vhpht_dd0803af-c8ee-4ae0-8082-1c4e47742909/kuadrant-console-plugin/0.log"
Oct 31 04:24:49.002055 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:24:49.002021    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:24:49.209164 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:49.209135    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-275tg_a5ebe756-cf88-441b-aba4-51bb5f9d1922/manager/0.log"
Oct 31 04:24:49.813740 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:49.813712    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7_c5de1f7a-3665-4f3c-9f23-3febe52c88d0/istio-proxy/0.log"
Oct 31 04:24:50.034969 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:50.034943    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-cx854_70b3d31c-a1ec-45db-97e5-cf9d1a0f227c/discovery/0.log"
Oct 31 04:24:51.004451 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:51.004416    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-dqwsm_84553d7f-51d4-4b72-b2dd-ee522e5413ad/istio-proxy/0.log"
Oct 31 04:24:51.256240 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:51.256153    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-jgmq4_92da6b1c-2d25-4e59-b877-a9b3b265ab3f/istio-proxy/0.log"
Oct 31 04:24:52.033424 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:52.033389    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-lcwvs_3d7daa66-ba76-4d1a-9de8-2a68928f22b7/istio-proxy/0.log"
Oct 31 04:24:53.289220 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:53.289195    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-666dbc6559-mnn8v_c539e6bf-2bd3-482c-9881-5c3fd79143e2/istio-proxy/0.log"
Oct 31 04:24:53.761953 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:53.761906    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-2z9q8_40e3764e-c5f1-4c75-b49e-13d815a622d2/payload-processing/0.log"
Oct 31 04:24:54.626477 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:54.626453    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-x7gqb_2b7caa8c-a0aa-4fbd-8918-288e39fc4f56/main/0.log"
Oct 31 04:24:54.843012 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:54.842984    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-nbrhn_98f65062-e7db-4f54-9d76-6addebbebae6/main/0.log"
Oct 31 04:24:55.069670 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:55.069641    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-xl2pv_a3767bbe-f08f-47b5-84eb-f01ebf697a41/main/0.log"
Oct 31 04:24:55.303080 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:55.303036    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-dsdwd_be4ef64f-41b3-4ec7-9408-bbbb03f85575/main/0.log"
Oct 31 04:24:55.545627 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:55.545593    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8lxd7r_7523be11-f92d-47cc-9309-998e5e4caa6b/main/0.log"
Oct 31 04:24:55.778417 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:55.778391    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-s6nfq_41c075e6-b57b-4e72-ad58-59450e64c8a3/main/0.log"
Oct 31 04:24:56.039954 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:24:56.039879    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-tq7j5_5c66fbd6-0415-4cb4-a9ca-fdfc437c55f7/main/0.log"
Oct 31 04:25:19.001424 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:25:19.001387    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:25:28.980195 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:25:28.980168    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-2z9q8_40e3764e-c5f1-4c75-b49e-13d815a622d2/payload-processing/0.log"
Oct 31 04:25:29.204567 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:25:29.204539    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-2z9q8_40e3764e-c5f1-4c75-b49e-13d815a622d2/payload-processing/0.log"
Oct 31 04:25:49.002731 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:25:49.002690    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:26:08.096353 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:08.096326    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-2z9q8_40e3764e-c5f1-4c75-b49e-13d815a622d2/payload-processing/0.log"
Oct 31 04:26:08.308090 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:08.308061    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-2z9q8_40e3764e-c5f1-4c75-b49e-13d815a622d2/payload-processing/0.log"
Oct 31 04:26:18.268511 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:18.268430    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7nb9d_8d251d9b-c5e7-46e7-b5ab-524b8171d83b/global-pull-secret-syncer/0.log"
Oct 31 04:26:19.001863 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:26:19.001825    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:26:21.358471 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:21.358401    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kqgxt_ff0da67d-c75e-4487-ba6a-b85e04a496d9/konnectivity-agent/0.log"
Oct 31 04:26:22.179035 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:22.179006    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-255.us-west-1.compute.internal_d21c869370d277ce1b45bad9863021dc/haproxy/0.log"
Oct 31 04:26:27.568003 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.567965    4447 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn"]
Oct 31 04:26:27.568717 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.568702    4447 cpu_manager.go:414] "containerMap: removing container" podUID="db7f8389-426b-4d60-ab96-a08a3cbbd391" containerName="revoke-keys"
Oct 31 04:26:27.568787 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.568720    4447 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db7f8389-426b-4d60-ab96-a08a3cbbd391" containerName="revoke-keys"
Oct 31 04:26:27.568863 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.568853    4447 memory_manager.go:365] "RemoveStaleState removing state" podUID="db7f8389-426b-4d60-ab96-a08a3cbbd391" containerName="revoke-keys"
Oct 31 04:26:27.572619 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.572600    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn"
Oct 31 04:26:27.621623 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.621596    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-4bnf2_9670f62f-e977-4370-951a-2a7bcb9209ac/manager/0.log"
Oct 31 04:26:27.625587 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.625562    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xr5kx\"/\"openshift-service-ca.crt\""
Oct 31 04:26:27.625732 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.625590    4447 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xr5kx\"/\"default-dockercfg-mqfbg\""
Oct 31 04:26:27.670580 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.670547    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfwbl\" (UniqueName: \"kubernetes.io/projected/54d99836-6c28-4716-bf0b-1cb8c9efb795-kube-api-access-dfwbl\") pod \"perf-node-gather-daemonset-f2dnn\" (UID: \"54d99836-6c28-4716-bf0b-1cb8c9efb795\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn"
Oct 31 04:26:27.670580 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.670585    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/54d99836-6c28-4716-bf0b-1cb8c9efb795-proc\") pod \"perf-node-gather-daemonset-f2dnn\" (UID: \"54d99836-6c28-4716-bf0b-1cb8c9efb795\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn"
Oct 31 04:26:27.670847 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.670610    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54d99836-6c28-4716-bf0b-1cb8c9efb795-sys\") pod \"perf-node-gather-daemonset-f2dnn\" (UID: \"54d99836-6c28-4716-bf0b-1cb8c9efb795\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn"
Oct 31 04:26:27.670968 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.670943    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/54d99836-6c28-4716-bf0b-1cb8c9efb795-podres\") pod \"perf-node-gather-daemonset-f2dnn\" (UID: \"54d99836-6c28-4716-bf0b-1cb8c9efb795\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn"
Oct 31 04:26:27.671049 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.670992    4447 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/54d99836-6c28-4716-bf0b-1cb8c9efb795-lib-modules\") pod \"perf-node-gather-daemonset-f2dnn\" (UID: \"54d99836-6c28-4716-bf0b-1cb8c9efb795\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn"
Oct 31 04:26:27.675331 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.675307    4447 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xr5kx\"/\"kube-root-ca.crt\""
Oct 31 04:26:27.680197 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.680171    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn"]
Oct 31 04:26:27.772021 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.771977    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/54d99836-6c28-4716-bf0b-1cb8c9efb795-podres\") pod \"perf-node-gather-daemonset-f2dnn\" (UID: \"54d99836-6c28-4716-bf0b-1cb8c9efb795\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn"
Oct 31 04:26:27.774154 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.772033    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/54d99836-6c28-4716-bf0b-1cb8c9efb795-lib-modules\") pod \"perf-node-gather-daemonset-f2dnn\" (UID: \"54d99836-6c28-4716-bf0b-1cb8c9efb795\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn"
Oct 31 04:26:27.774154 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.772074    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfwbl\" (UniqueName: \"kubernetes.io/projected/54d99836-6c28-4716-bf0b-1cb8c9efb795-kube-api-access-dfwbl\") pod \"perf-node-gather-daemonset-f2dnn\" (UID: \"54d99836-6c28-4716-bf0b-1cb8c9efb795\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn"
Oct 31 04:26:27.774154 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.772099    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/54d99836-6c28-4716-bf0b-1cb8c9efb795-proc\") pod \"perf-node-gather-daemonset-f2dnn\" (UID: \"54d99836-6c28-4716-bf0b-1cb8c9efb795\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn"
Oct 31 04:26:27.774154 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.772111    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/54d99836-6c28-4716-bf0b-1cb8c9efb795-podres\") pod \"perf-node-gather-daemonset-f2dnn\" (UID: \"54d99836-6c28-4716-bf0b-1cb8c9efb795\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn"
Oct 31 04:26:27.774154 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.772133    4447 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54d99836-6c28-4716-bf0b-1cb8c9efb795-sys\") pod \"perf-node-gather-daemonset-f2dnn\" (UID: \"54d99836-6c28-4716-bf0b-1cb8c9efb795\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn"
Oct 31 04:26:27.774154 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.772182    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54d99836-6c28-4716-bf0b-1cb8c9efb795-sys\") pod \"perf-node-gather-daemonset-f2dnn\" (UID: \"54d99836-6c28-4716-bf0b-1cb8c9efb795\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn"
Oct 31 04:26:27.774154 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.772180    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/54d99836-6c28-4716-bf0b-1cb8c9efb795-lib-modules\") pod \"perf-node-gather-daemonset-f2dnn\" (UID: \"54d99836-6c28-4716-bf0b-1cb8c9efb795\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn"
Oct 31 04:26:27.774154 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.772193    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/54d99836-6c28-4716-bf0b-1cb8c9efb795-proc\") pod \"perf-node-gather-daemonset-f2dnn\" (UID: \"54d99836-6c28-4716-bf0b-1cb8c9efb795\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn"
Oct 31 04:26:27.833086 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.833008    4447 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfwbl\" (UniqueName: \"kubernetes.io/projected/54d99836-6c28-4716-bf0b-1cb8c9efb795-kube-api-access-dfwbl\") pod \"perf-node-gather-daemonset-f2dnn\" (UID: \"54d99836-6c28-4716-bf0b-1cb8c9efb795\") " pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn"
Oct 31 04:26:27.888141 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:27.888108    4447 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn"
Oct 31 04:26:28.041940 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:28.041878    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-mtv6q_f4fc7df3-3489-458c-971e-7f2717626680/manager/0.log"
Oct 31 04:26:28.261879 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:28.261849    4447 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn"]
Oct 31 04:26:28.263060 ip-10-0-134-255 kubenswrapper[4447]: W1031 04:26:28.263017    4447 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod54d99836_6c28_4716_bf0b_1cb8c9efb795.slice/crio-404fbe9c5b263532d3e8450d35ae13fc000ccce7cb4d97a93c7d57f2092ad755 WatchSource:0}: Error finding container 404fbe9c5b263532d3e8450d35ae13fc000ccce7cb4d97a93c7d57f2092ad755: Status 404 returned error can't find the container with id 404fbe9c5b263532d3e8450d35ae13fc000ccce7cb4d97a93c7d57f2092ad755
Oct 31 04:26:28.389036 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:28.389008    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn" event={"ID":"54d99836-6c28-4716-bf0b-1cb8c9efb795","Type":"ContainerStarted","Data":"404fbe9c5b263532d3e8450d35ae13fc000ccce7cb4d97a93c7d57f2092ad755"}
Oct 31 04:26:28.462282 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:28.462254    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vhpht_dd0803af-c8ee-4ae0-8082-1c4e47742909/kuadrant-console-plugin/0.log"
Oct 31 04:26:29.396785 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:29.396744    4447 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn" event={"ID":"54d99836-6c28-4716-bf0b-1cb8c9efb795","Type":"ContainerStarted","Data":"c8eef9f7e512cea55416da1006a3a5f81fe53dbb15e40697018e022c87852138"}
Oct 31 04:26:29.397218 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:29.396844    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn"
Oct 31 04:26:29.513724 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:29.513671    4447 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn" podStartSLOduration=2.513656993 podStartE2EDuration="2.513656993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:26:29.512370446 +0000 UTC m=+4015.023352826" watchObservedRunningTime="2026-10-31 04:26:29.513656993 +0000 UTC m=+4015.024639370"
Oct 31 04:26:29.831652 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:29.831618    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-275tg_a5ebe756-cf88-441b-aba4-51bb5f9d1922/manager/0.log"
Oct 31 04:26:32.418834 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:32.418805    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_17f3e0fe-a1e5-44ec-a293-a557799c7da2/alertmanager/0.log"
Oct 31 04:26:32.837713 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:32.837684    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_17f3e0fe-a1e5-44ec-a293-a557799c7da2/config-reloader/0.log"
Oct 31 04:26:33.268414 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:33.268387    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_17f3e0fe-a1e5-44ec-a293-a557799c7da2/kube-rbac-proxy-web/0.log"
Oct 31 04:26:33.687040 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:33.686965    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_17f3e0fe-a1e5-44ec-a293-a557799c7da2/kube-rbac-proxy/0.log"
Oct 31 04:26:34.104818 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:34.104791    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_17f3e0fe-a1e5-44ec-a293-a557799c7da2/kube-rbac-proxy-metric/0.log"
Oct 31 04:26:34.526594 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:34.526565    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_17f3e0fe-a1e5-44ec-a293-a557799c7da2/prom-label-proxy/0.log"
Oct 31 04:26:34.945864 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:34.945787    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_17f3e0fe-a1e5-44ec-a293-a557799c7da2/init-config-reloader/0.log"
Oct 31 04:26:35.414258 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:35.414223    4447 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xr5kx/perf-node-gather-daemonset-f2dnn"
Oct 31 04:26:35.794454 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:35.794411    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-h4gsb_213aee5f-38c2-41ce-8790-e37ec9437e10/kube-state-metrics/0.log"
Oct 31 04:26:36.206789 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:36.206765    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-h4gsb_213aee5f-38c2-41ce-8790-e37ec9437e10/kube-rbac-proxy-main/0.log"
Oct 31 04:26:36.624290 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:36.624212    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-h4gsb_213aee5f-38c2-41ce-8790-e37ec9437e10/kube-rbac-proxy-self/0.log"
Oct 31 04:26:37.049084 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:37.049053    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-846c75f654-4rxqg_2cbb3341-678d-484b-b2c1-2112c6ae4f34/metrics-server/0.log"
Oct 31 04:26:37.470843 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:37.470814    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-mtnb7_3ca103aa-c5e7-4753-b429-159893db3ab6/monitoring-plugin/0.log"
Oct 31 04:26:37.621518 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:37.621484    4447 scope.go:122] "RemoveContainer" containerID="b70405c6253fe07017c103b1a6b6ec8f42f447eb8e176ccfec4cc66e11df56d1"
Oct 31 04:26:39.174645 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:39.174614    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8gkd5_311bf3de-3730-4e0b-9872-192c9bde06cf/node-exporter/0.log"
Oct 31 04:26:39.600591 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:39.600559    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8gkd5_311bf3de-3730-4e0b-9872-192c9bde06cf/kube-rbac-proxy/0.log"
Oct 31 04:26:40.022766 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:40.022739    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8gkd5_311bf3de-3730-4e0b-9872-192c9bde06cf/init-textfile/0.log"
Oct 31 04:26:45.969733 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:45.969701    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-r4qz8_09708991-5135-48b2-9615-1b40d5375e63/prometheus-operator/0.log"
Oct 31 04:26:46.390456 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:46.390381    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-r4qz8_09708991-5135-48b2-9615-1b40d5375e63/kube-rbac-proxy/0.log"
Oct 31 04:26:47.240089 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:47.240057    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-666c5cbddb-69wsv_66185435-76f9-4541-b002-0f3f7bf43f52/telemeter-client/0.log"
Oct 31 04:26:47.659172 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:47.659092    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-666c5cbddb-69wsv_66185435-76f9-4541-b002-0f3f7bf43f52/reload/0.log"
Oct 31 04:26:48.077545 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:48.077516    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-666c5cbddb-69wsv_66185435-76f9-4541-b002-0f3f7bf43f52/kube-rbac-proxy/0.log"
Oct 31 04:26:48.502741 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:48.502707    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-784c68f98c-wjj6h_a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac/thanos-query/0.log"
Oct 31 04:26:48.921368 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:48.921292    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-784c68f98c-wjj6h_a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac/kube-rbac-proxy-web/0.log"
Oct 31 04:26:49.003050 ip-10-0-134-255 kubenswrapper[4447]: E1031 04:26:49.003018    4447 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbl5h" containerName="extractor"
Oct 31 04:26:49.339955 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:49.339929    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-784c68f98c-wjj6h_a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac/kube-rbac-proxy/0.log"
Oct 31 04:26:49.758352 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:49.758324    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-784c68f98c-wjj6h_a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac/prom-label-proxy/0.log"
Oct 31 04:26:50.178076 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:50.177993    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-784c68f98c-wjj6h_a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac/kube-rbac-proxy-rules/0.log"
Oct 31 04:26:50.494757 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:50.494721    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-79f98cb86f6j9q7_c5de1f7a-3665-4f3c-9f23-3febe52c88d0/istio-proxy/0.log"
Oct 31 04:26:50.603445 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:50.603415    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-784c68f98c-wjj6h_a5d0bc3d-fc8f-4d56-a568-b2bb1b6b4aac/kube-rbac-proxy-metrics/0.log"
Oct 31 04:26:51.556097 ip-10-0-134-255 kubenswrapper[4447]: I1031 04:26:51.556059    4447 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-cx854_70b3d31c-a1ec-45db-97e5-cf9d1a0f227c/discovery/0.log"